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Dear  Reader: 


This  issue  of  the  Review  is  devoted  to  the  Acquisition  Management  area  of 
Test  and  Evaluation.  -5 


The  problems  a.s.sociated  with  Test  and  Evaluation  are  many  and  are  very 
critical  at  every  milestone  of  the  acquisition  process. 

As  the  costs  of  testing  increase,  the  Project  Manager  has  a very  difficult  task 
in  resource  management  against  the  requirements  for  technical  decision.s,  the 
rigidity  of  schedule  and  funding  constraints.  The  adequacy  of  the  data  base, 
the  extent  of  credible  instrumentation,  and  the  use  of  simulations  vs  actual 
tests,  all  play  a major  role. 

^j;XThe  need  for  Test  and  Evaluation  planning  early  in  all  phases  of  the 
acquisition  process  cannot  be  overstre,ssed.  It  is  of  particular  importance  to 
determine  the  criteria  for  success  or  failure  well  before  the  tests  are  planned 
and  conducted  so  that  evaluation  of  the  results  will  have  integrity.  One  of  our 
greatest  difficulties  is  insuring  the  testing  environment  is  as  close  to  the 
operational  environment  asywe  can  make  it.  Failure  to  do  so  may  adversely 
affect  the  ultimate  customer — the  soldier,  sailor,  marine  or  airman. 

' ^ Ijjjj  issu^Test  and  Evaluation  is  addressed  from  different  perspectives. 

Viewpoints  vary  as  do  the  reported  approaches  to  management.-Jhis  issue 
was  designed  to  stimulate  dialogue  and  improve  our  Test  and  E^luation 
plans  and  processes  while,  most  importantly,  achieving  credible  resultsN^ 


R.  G.  FREEMAN  III 
Rear  Admiral,  USN 
Commandant 
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In  consonance  with  the  intentions  of  the  Secretary  of  Defense  to  reorganize  his  depart- 
ment, the  functions  and  scope  of  the  former  Deputy  Director  for  Test  and  Evaluation 
(DD(T&E))  are  undergoing  revision.  The  principal  change  involves  the  delegation  of  re- 
sponsibility for  operational  testing  to  the  Undersecretary  for  Program  Analysis  and  Eval- 
uation (PA&E).  The  Test  and  Evaluation  organization  in  the  Office  of  the  Undersecretary 
of  Defense  (Research  & Engineering)  retains  primary  responsibility  for  developmental 
testing  as  well  as  test  resources  and  remains  the  focal  point  within  the  Office  of  the  Sec- 
retary of  Defense  (OSD)  for  all  Test  and  Evaluation  matters.  This  organizational  realign- 
ment, coupled  with  a recent  OSD-wide  staff  reduction,  has  the  effect  of  reducing  the 
former  DD(T&E)  staff  by  two-thirds.  These  changes  in  organization  and  functions  will 
be  reflected  in  a forthcoming  revision  to  Department  of  Defense  Directive,  Number 
500G.3;  however,  they  do  not  affect  the  principles  of  test  and  evaluation  stated  in  the  ar- 
ticles in  this  issue. 
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TEST  AND  EVALUATION  IN 
THE  DEPARTMENT  OF  DEFENSE 

by 

Lieutenant  General  Walter  E.  Lotz,  USA  (Ret) 

Former  Deputy  Director  of  Defense  Research  and  Engineering 
(Test  & Evaluation) 


Test  and  Evaluation  has  played  an  important  part  in  the  significant  improvements  made  in 
Defense  systems  acquisition  management  during  the  1970*s.  Prior  to  this  decade,  emphasis  was 
placed  on  the  “total  package  procurement”  concept  in  which  a contract  would  be  let  for  a 
complete  system  development  and  procurement  program  after  an  initial  paper  definition  ^hase. 
The  theory  was  that  if  a program  was  sufficiently  defined  in  the  beginning,  a contractor  could 
be  expected  to  deliver  the  required  product  at  a predetermined  cost.  Unfortunately,  the  total 
package  procurement  concept  did  not  work  well  in  practice  largely  because  of  overoptimistic 
cost  and  performance  estimates  and  inaccurate  and  unverifled  inputs  to  initial  definition 
phases.  The  result  was  that  many  programs  managed  under  this  concept  experienced  large  cost 
overruns  and  signiHcant  performance  deficiencies. 

POLICY  EVOLVEMENT 

About  1970,  many  groups,  including  the 
Blue  Ribbon  Defense  Panel,  the  Commission 
on  Government  Procurement  and  the  Defense 
Science  Board,  began  to  recognize  the  deficien- 
cies of  then  current  defense  acquisition  manage- 
ment practices.  Partly  as  a result  of  their  rec- 
ommendations, new  policies  evolved  that  em- 
phasized demonstrated  performance  as  the  pac- 
ing function  for  defense  programs.  The  key 
feature  of  the  new  policies  is  the  periodic  review 
of  acquisition  programs  at  critical  milestones. 

During  these  periodic  reviews,  measured  pro- 
gram progress  is  compared  with  program  goals 
and  objectives  and  a decision  made  to  continue, 
reorient  or  cancel  the  program.  A significant 
improvement  offered  by  the  new  policies  rela- 
tive to  those  that  had  prevailed  in  the  past  is 
that  program  deviations  can  be  detected 
quickly  and  corrective  actions  can  be  taken 
early. 

To  be  effective,  these  new  acquisition  poli- 
cies are  heavily  dependent  upon  reliable  and 
accurate  measurement  of  program  progress  as 


compared  to  program  objectives.  Test  and  eval- 
uation, the  primary  means  for  making  such 
measurements,  became  the  cornerstone  of  the 
new  acquisition  policies  and  received  emphasis 
in  the  implementation  of  these  policies.  One 
significant  result  of  this  emphasis  was  the  estab- 
lishment of  the  Office  of  the  Deputy  Director 
(Test  and  Evaluation)  in  the  Pentagon’s  Direc- 
torate of  Defense  Research  and  Engineering.  In 
addition,  each  Service  was  required  to  establish 
independent  test  agencies  and  strong  headquar- 
ters staff  focal  points  to  assist  in  conducting 
required  test  and  evaluation.  As  a further  step, 
appropriate  DOD  and  Service  directives  were 
promulgated  to  spell  out  the  increased  empha- 
sis on  test  and  evaluation  in  the  acquisition 
process. 

TEST,  EVALUATION  AND 
SYSTEMS  ACQUISITION 

There  are  two  principal  kinds  of  test  and 
evaluation  conducted  in  the  systems  acquisition 
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process — Development  Test  and  Evaluation 
(DT&E)  and  Operational  Test  and  Evaluation 
(OT&E).  Development  Test  and  Evaluation  is 
that  testing  conducted  by,  or  under  the  supervi- 
sion of,  the  development  agency  to  evaluate 
technical  performance  of  prototype  equipment. 
This  testing  is  generally  conducted  by  skilled 
technicians  and  engineers  under  carefully  con- 
trolled conditions.  Operational  Test  and  Evalu- 
ation is  that  testing  conducted  by  military  per- 
sonnel to  determine  the  degree  to  which  new 
equipment  fulfills  military  operational  require- 
ments. The  Operational  Test  and  Evaluation  is 
conducted  under  conditions  that  duplicate  as 
closely  as  possible  the  environment  expected  in 
field  operations.  Further,  Operational  Test  and 
Evaluation  is  conducted  on  early  production 
models  as  well  as  on  research  and  development 
prototypes  of  new  equipment. 


FUNCTIONS 

Development  Test  and  Evaluation  serves 
two  important  functions  in  the  systems  acquisi- 
tion process.  First  Development  Test  and  Eval- 
uation assists  in  the  actual  design  and  develop- 
ment of  systems.  In  this  role  Development  Test 
and  Evaluation  is  an  integral  part  of  the  normal 
development  process  in  which  initial  designs 
are  converted  to  hardware.  The  hardware  is 
tested,  deficiencies  are  noted,  and  the  design  is 
modified  as  necessary.  This  process  is  iterated 
until  the  system  hardware  reaches  a final  design 
configuration.  The  development  testing  con- 
ducted in  this  iterative  process  can  be  thought 
of  as  providing  the  essential  “feed  back”  infor- 
mation required  to  proceed  from  one  step  to  the 
next  in  converting  an  initial  paper  design  into 
fully  developed  hardware. 


A second  important  function  of  Develop- 
ment Test  and  Evaluation  is  to  provide  infor- 
mation on  the  progress  of  new  system  develop- 
ment. The  progress  is  ascertained  by  comparing 
measured  system  performance  with  a time- 
phased  set  of  goals  and  objectives  which  have 
been  established  for  the  program.  These  time- 
phased  goals  and  objectives  are  based  on  the 
premise  that  a system  evolves  or  “matures” 
during  development.  In  this  maturing  process 


certain  characteristics,  e.g.,  reliability,  will  typi- 
cally improve,  while  other  characteristics,  e.g., 
system  weight,  may  typically  deteriorate  or 
remain  static.  If  we  are  sufficiently  familiar 
with  this  maturing  process,  we  can  determine 
what  performance  levels  should  be  attained  by 
the  system  at  various  stages  of  development  to 
reach  the  final,  required  levels  of  performance. 
Development  tests  then  can  be  performed  at 
frequent  points  in  the  program  and  provide  a 
decision  maker  with  information  on  the  rate  of 
maturation  of  the  system.  Additionally,  a pre- 
diction of  final  system  performance  can  be 
provided  by  comparing  performance  level 
progress  with  similar  systems. 

The  functions  served  by  Operational  Test 
and  Evaluation  in  the  acquisition  process  are 
somewhat  similar  to  those  of  Development  Test 
and  Evaluation.  To  the  extent  that  Operational 
Test  and  Evaluation  is  performed  on  research 
and  development  prototypes,  it  aids  system 
design  and  development  by  the  early  detection 
and  correction  of  operational  deficiencies.  The 
detection  and  correction  of  deficiencies  during 
development  provides  significant  time  and  cost 
advantages  by  procluding  the  necessity  for  re- 
trofitting operational  systems. 

Like  Development  Test  and  Evaluation, 
Operational  Test  and  Evaluation  also  provides 
essential  information  for  decisionmaking  by 
comparing  system  operational  performance 
with  program  goals  and  objectives.  Since 
Operational  Test  and  Evaluation  conducted  be- 
fore system  production  involves  testing  of  pro- 
totypes, to  predict  final  system  operational  per- 
formance test  results  must  be  extrapolated  as  in 
Development  Test  and  Evaluation.  The  degree 
of  extrapolation  required  will  depend  upon  the 
stage  of  system  development  and  the  realism  of 
the  simulated  test  environment. 

Although  performing  the  same  generic 
functions  in  the  acquisition  process.  Opera- 
tional Test  and  Evaluation  and  Development 
Test  and  Evaluation  have  distinctive  differ- 
ences. Operational  Test  and  Evaluation  is  con- 
cerned more  with  evaluating  purely  operational 
as  opposed  to  technical  factors  and  gives  more 
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emphasis  to  predicting  future  performance 
rather  than  evaluating  current  performance. 
With  regard  to  the  latter  distinction,  recognize 
that  there  is  normally  only  one  Operational 
Test  and  Evaluation  conducted  on  a research 
and  development  prototype,  and  single  point 
projections  are  error-prone.  A combination  of 
Operational  Test  and  Evaluation  with  Develop- 
ment Test  and  Evaluation  provides  the  most 
effective  means  to  predict  mature  performance. 


"TRY  BEFORE  BUY" 

While  much  has  been  made  of  the  “try 
before  buy”  aspect  of  test  and  evaluation,  this 
aspect  is  a considerable  oversimplification  of 
the  current  role  of  test  and  evaluation  in  the 
systems  acquisition  process.  Although  there  are 
other  significant  considerations,  most  impor- 
tant decisions  in  the  acquisition  process  will 
normally  require  information  relating  to  two 
basic  questions: 

• What  is  current  program  status?  and, 

• What  is  the  likely  outcome  of  the 
program? 

As  the  preceding  discussion  indicates,  this  is 
exactly  the  kind  of  information  test  and  evalua- 
tion provides.  It  is  apparent  that  test  and  evalu- 
ation is  quite  important  not  only  for  major 
production  decisions  but  for  most  other  deci- 
sions made  during  the  life  of  an  acquisition 
program. 

For  major  defense  programs,  test  and  eval- 
uation impacts  on  the  acquisition  process  pri- 
marily at  the  Office,  Secretary  of  Defense  level 
through  the  operation  of  the  Defense  Systems 
Acquisition  Review  Council  (DSARC).  The 
DSARC  is  an  advisory  group  to  the  Secretary 
of  Defense  and  is  chaired  by  the  DOD  Acquisi- 
tion Executive.  Membership  is  made  up  from 
those  who  head  major  DOD  staff  offices.  When 
a defense  program  reaches  a major  milestone, 
the  DSARC  meets  to  consider  whether  the  pro- 
gram should  be  advanced  to  its  next  phase.  The 


DSARC  then  provides  its  recommendations  di- 
rectly to  the  Secretary  of  Defense  who  makes 
the  final  decision.  Although  not  a member  of 
the  DSARC,  the  Deputy  Undersecretary  of  De- 
fense, Research  and  Engineering  (Test  and 
Evaluation)  DUSD  R&E(T&E)  plays  an  im- 
portant role  in  the  DSARC  process.  Prior  to  a 
DSARC  he  submits  to  each  DSARC  member 
his  detailed  assessment  of  test  and  evaluation 
conducted  to  date  or  planned  for  the  future  on 
the  program  under  consideration.  The  Deputy 
Undersecretary  of  Defense,  Research  and  Engi- 
neering (Test  and  Evaluation)  also  participates 
in  the  actual  DSARC  discussions  leading  to  a 
DSARC  recommendation  to  the  Secretary  of 
Defense.  After  the  DSARC  meeting,  the  Dep- 
uty Director  submits  a test  and  evaluation  as- 
sessment directly  to  the  Secretary  of  Defense 
for  use  when  considering  the  recommendations 
of  the  DSARC.  The  effect  of  these  procedures  is 
to  insure  that  systems  acquisition  decisions  are 
based  on  a complete  and  independent  assess- 
ment of  a program’s  test  and  evaluation  status. 

Recent  changes  to  Defense  Acquisition 
policies  delegate  DSARC-type  functions  for 
some  systems  to  the  Service  Acquisition  Re- 
view Councils.  In  these  cases  test  and  evalua- 
tion plays  the  same  important  role. 

Although  sometimes  difficult  to  achieve, 
independence  and  objectivity  are  quite  impor- 
tant to  the  test  and  evaluation  assessments 
provided  for  decisionmaking.  Since  most  De- 
velopment Test  and  Evaluation  is  an  integral 
part  of  the  systems  design  and  development 
process,  it  is  rightly  conducted  or  controlled  by 
the  developing  agency.  Development  agencies 
are,  traditionally,  “success  oriented”  and  as 
such  do  not  always  give  adequate  weight  to  the 
problems  and  deficiencies  that  sometimes  arise 
in  a program.  Similarly,  operational  testing 
cannot  be  conducted  without  the  extensive  par- 
ticipation of  user  personnel  since  user  personnel 
are  needed  to  achieve  adequate  realism.  Using 
commands,  however,  may  tend  to  minimize 
system  deficiencies  in  seeking  early  deployment 
of  a system,  or  alternatively  may  seek  refine- 
ments in  a system  which,  although  “nice  to 
have,”  are  not  essential  to  mission  perfor- 
mance. To  the  extent  possible,  the  effects  of 
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such  potential  biases  must  be  reduced  by  pro- 
viding independent  assessment  and  reporting  of 
testing.  Such  independence  is  achieved  at  the 
Office  of  the  Secretary  of  Defense  level  by  the 
DD(T&E)  reporting  test  and  evaluation  assess- 
ments directly  to  the  Secretary  of  Defense. 
Similarly,  within  the  Services,  the  independent 
test  agencies  report  directly  to  the  Chief  of 
Service  rather  than  through  either  a developer 
or  user  chain  of  command.  These  independent 
reporting  arrangements  are  specifically  re- 
quired by  current  DOD  directives. 

Since  1971  considerable  emphasis  has 
been  placed  on  Operational  Test  and  Evalua- 
tion, particularly  that  portion  of  operational 
te.=ting  that  immediately  precedes  a decision  to 
produce  and  deploy  a weapon  system.  Opera- 
tional testing  provides  an  estimate  of  how  the 
system  will  perform  in  the  operational  environ- 
ment. Short  of  actual  warfare,  operational  test- 
ing is  the  ultimate  measure  of  an  acquisition 
program’s  output  and  as  such  will  normally 
receive  considerable  emphasis  when  decisions 
are  to  be  reached  about  resource  commitments 
to  specific  weapon  systems. 


Effective  operational  testing  requires  as 
much  test  realism  as  possible  consistent  with 
resource  constraints  and  hardware  status.  This 
means  the  use  of  representative  models  of 
production  systems,  the  participation  jf  typical 
operational  personnel  in  testing  and,  accurate 
simulation  of  both  the  threat  and  the  physical 
environment.  Accurate  simulation  of  the  opera- 
tional environment  may  sometimes  require 
two-sided  testing  involving  simulated  friendly 
and  aggressor  forces. 


There  are  several  other  factors  that  must 
be  considered  in  providng  for  effective  opera- 
tional testing.  Operational  test  designs  must 
allow  for  the  exercise  of  systems  over  a reasona- 
ble range  of  operational  conditions.  We  cannot 
fall  into  the  trap  of  reaching  a conclusion  on  the 
overall  operational  effectiveness  of  a system 
based  on  testing  under  a single  set  of  conditions 
that  may  represent  only  a small  percentage  of 
likely  tactical  situations.  Similarly,  the  scope  of 


the  testing  must  allow  for  all  significant  interac- 
tion between  the  system  being  tested  and  other 
systems  with  which  it  must  function  in  the 
operational  environment.  Finally,  it  is  impor- 
tant that  objective  measurements  of  a system’s 
performance  in  simulated  operational  employ- 
ment be  made  without  compromise  to  realism. 
In  the  last  few  years  we  have  made  great  strides 
in  developing  instrumentation  that  will  allow  us 
to  make  such  objective  measurements  without 
interfering  in  actual  test  conduct. 


Certainly  not  the  least  important  aspect  of 
effective  utilization  of  test  and  evaluation  in  the 
acquisition  process  is  the  establishment  of 
meaningful  program  goals  and  objectives  whose 
attainment  is  to  be  demonstrated  by  test.  The 
task  is  difficult  and  requires  the  interaction  of 
many  segments  of  the  Defense  community.  For 
example,  the  operational  forces  have  to  provide 
information  on  critical  capabilities  needed  to 
meet  particular  operational  requirements.  The 
system  development  commands  must  provide 
inputs  on  resource  constraints  and  critical  tech- 
nical parameters.  The  test  community  must 
provide  inputs  on  test  feasibility  and  test  facility 
requirements.  Lastly,  the  analytical  community 
must  contribute  by  helping  define  measures  of 
effectiveness,  confidence  levels,  and  data  re- 
quirements. If  the  efforts  of  these  various 
groups  can  be  integrated  effectively,  we  can  be 
successful  in  defining  proper  goals  and  objec- 
tives against  which  program  progress  can  be 
measured. 


Test  and  evaluation  has  become  an  essen- 
tial and  integral  part  of  the  systems  acquisition 
process.  We  recognize  that  there  are  other 
improvements  which  can  and  should  be  made 
to  enhance  test  and  evaluation  effectiveness. 


One  such  improvement  is  to  begin  limited 
operational  testing  earlier  in  the  acquisition 
cycle  of  a weapon  system.  In  the  early  1970’s 
almost  exclusive  emphasis  was  placed  on  con- 
ducting Operational  Test  and  Evaluation  just 
prior  to  the  major  production  decision.  This 
was  because  of  the  large  funding  commitment 
resulting  from  the  production  decision.  Lately 
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we  have  realized  that  earlici  Operational  Test 
and  Evaluation  may  be  needed  for  four  reasons; 


• In  considering  systems  concepts  dur- 
ing the  early  stages  of  a program,  we 
need  to  know  if  these  concepts  are 
technically  viable  or  tactically  sound. 

• To  have  any  impact  on  the  research 
and  development  phase  of  a program, 
Operational  Test  and  Evaluation 
should  be  conducted  much  earlier  than 
just  before  the  major  production  deci- 
sion since  by  then  most  research  and 
development  resources  will  have  been 
expended. 


• Cost  and  schedule  impact  can  be  mini- 
mized by  early  detection  and  correc- 
tion of  operational  deficiencies. 


• A recent  trend  toward  initiating 
limited  production  before  the  major 
production  decision  has  resulted  in  sig- 
nificant commitments  of  procurement 
funds  prior  to  the  major  production  de- 
cision. 


For  these  reasons,  we  need  to  place  more  em- 
phasis on  early  operational  testing.  While  we 
cannot  expect  as  much  realism  in  early  opera- 
tional testing  as  can  be  achieved  at  a later 
program  stage,  the  information  obtained  will 
provide  significant  improvements  in  the  quality 
of  inputs  to  decisions  that  are  made  early  in  the 
acquisition  process. 

Another  improvement  of  considerable 
benefit  is  to  relate  test  and  evaluation  efforts  to 
the  actual  risks  involved  in  a particular  acquisi- 
tion program.  The  link  between  testing  and 
program  risk  is  obvious.  For  the  zero  risk  case 
we  would  probably  need  to  do  little  or  no  test 
and  evaluation  to  assist  the  decision  process. 
For  the  high  risk  case,  where  we  know  little 


about  possible  decision  outcomes,  we  might 
want  to  do  a great  deal  of  test  and  evaluation  to 
enhance  the  probability  of  making  the  “cor- 
rect” decision. 

The  implications  of  this  relationship  are 
clear.  We  need  to  expend  more  effort  in  defin- 
ing the  various  technical,  operational,  cost  and 
other  risks  that  exist  within  an  acquisition  pro- 
gram. We  need  to  encourage  decision  makers  to 
define  the  level  of  risk  that  can  be  tolerated  at 
various  decision  points.  Given  these  inputs,  we 
can  “tailor”  a test  and  evaluation  program  to 
better  fit  the  risk  situation  that  exists  at  a 
particular  decision  point.  The  result  will  be  a 
more  efficient  and  more  economical  test  and 
evaluation  program. 

Another  important  function  test  and  eval- 
uation can  perform  is  to  provide  an  adequate 
transfer  of  “lessons  learned”  to  the  system 
design  and  engineering  community.  Often  a 
deficiency  discovered  during  test  and  evalua- 
tion on  one  program  is  nearly  identical  to  that 
discovered  on  past  or  present  programs.  Test 
and  evaluation  can  play  a significant  role  in 
reducing  this  iteration.  An  example  is  provided 
by  a current  test  and  evaluation  program  in 
which  we  are  modifying  military  specifications 
and  handbooks  and  publishing  appropriate  de- 
sign guides  to  insure  that  design  deficiencies 
discovered  in  electromagnetic  susceptibility 
testing  of  one  system  are  not  repeated  in  other 
systems.  Be  expanding  such  efforts  to  capitalize 
on  lessons  learned  during  testing,  the  test  and 
evaluation  community  can  improve  efficiency 
and  the  end  products  of  the  weapon  systems 
acquisition  process. 


In  summary,  test  and  evaluation  has  be- 
come firmly  established  as  an  integral  part  of 
the  weapon  systems  acquisition  process.  We  in 
the  test  and  evaluation  community  are  now 
concentrating  on  improving  the  effectiveness  of 
test  and  evaluation  and  adapting  it  to  changing 
acquisition  strategies.  The  ultimate  payoff  of 
these  efforts  will  be  maximum  military  capabil- 
ity for  our  limited  defense  dollars. 
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The  Army  is  ever  changing.  World  War  II  veterans  express  shock  at  the  differences  be- 
tween the  brown  shoe  Army  and  the  All  Volunteer  Force.  Usually,  caustic  comments  refer  to 
the  modern  barracks,  higher  pay,  better  living  conditions  and  other  facets  of  the  “new”  Army. 
But,  similar  observations  might  be  made  when  these  veterans  look  at  the  changes  in  the  way  the 
Army  equips  modern  volunteers  and  makes  sure  Army  materiel  is  the  best  that  Defense  dollars 
can  buy. 

Materiel  test  and  evaluation  formerly  was  simple  and  less  complicated.  A weapon  or  piece 
of  equipment  was  examined  by  the  service  boards  for  engineering  quality,  fielded  for  experi- 
mental purposes  and  accepted  or  rejected  largely  on  the  field  commander’s  opinion  of  how  the 
item  performed  while  being  handled  by  the  troops. 

Today,  the  process  is  more  intricate.  Several  Army  commands  enter  the  test  and  evalua- 
tion picture  in  a variety  of  phases.  Decisions  involve  the  highest  levels  of  management  including 
Congress  and  the  White  House.  It  would  seem  that  development  and  deployment  of  materiel 
would  be  cumbersome,  more  costly  and,  understandably,  take  longer.  But  should  it? 


TRANSITION 

The  old  proverb,  “The  more  things 
change,  the  more  they  remain  the  same,”  is  true 
here.  While  there  are  many  more  players  in  the 
materiel  life  cycle  now,  the  basic  two,  the  Army 
and  industry,  remain — as  does  the  traditional 
and  common  goal  to  cut  production  costs. 
Working  together  on  new  approaches  to  acquir- 
ing materiel,  there  is  hope  that  these  two  part- 
ners can  achieve  some  mutually  beneficial  in- 
roads toward  shorter  and  less  expensive  devel- 
opment cycles,  while  maintaining  high  stan- 
dards of  quality. 

The  standard  manner  in  which  the  Army 
formerly  conducted  test  and  evaluation  had  to 
change.  Reduced  budgets  and  manpower  was 
one  reason.  Another  reason  was  that  too  much 
duplication  was  taking  place,  both  among  facil- 
ities and  tefiters.  Example;  it  took  13  years  to 
field  the  Redeye  Air  Defense  Missile.  Much  of 
that  time  was  used  to  put  the  item  through  a 
series  of  sequential  tests,  making  sure  it  would 
do  the  intended  job.  That  approach  is  no  longer 
affordable. 


Today,  new  requirements,  burgeoning 
costs  and  new  technology  require  shorter  devel- 
opment times.  With  high  costs,  layers  of  deci- 
sionmakers and  a complex  test  sequence,  there 
is  an  urgent  need  to  streamline.  Testing  is  a 
productive  area  in  which  to  start. 

Testing  has  become  the  key  to  the  develop- 
ment process  that  accelerates  as  testing  be- 
comes more  objective  and  uniform,  and  as 
schedules  more  completely  integrate  the  efforts 
of  the  many  participants.  We  need  an  approach 
that  insures  quality,  economy  and  flexibility. 
We  believe  we  are  getting  closer  to  it.  After  a 
difficult  metamorphosis,  the  Army  test  com- 
munity has  emerged  somewhat  streamlined  in 
its  test  and  evaluation  posture.  This  is  what  we 
look  for: 

• Shorter  tests. 

• Schedules  where  efforts  can  be  com- 
bined and  outlays  reduced. 
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• Ways  testers  can  work  with  industry 
and  developers  to  further  reduce  ex- 
penditures. 

Before  explaining  our  present  approach, 
let’s  look  at  how  test  and  evaluation  evolved. 

HISTORY 

Prior  to  1924,  when  the  Army  issued  regu- 
lations making  test  and  evaluation  integral  to 
the  procurement  of  weapons  and  equipment. 
Army-wide  purchasing  decisions  were  based  on 
an  officer  board’s  experience  with  a weapon  or 
second-hand  reports  from  the  field.  From  the 
Colonial  period  on,  soldiers  often  purchased 
weapons  on  their  own. 

Following  the  1924  decision  and  until 
World  War  II,  the  combat  arms  (infantry,  cav- 
alry, field  artillery,  coast  artillery)  requested  de- 
velopment of  new  equipment  and  determined 
characteristics.  Supply  arms  and  services  (engi- 
neer, signal,  ordnance,  chemical,  quartermas- 
ter, medical)  were  charged  with  conducting  de- 
velopment projects  within  their  respective 
spheres. 

Engineering  tests,  conducted  by  the  devel- 
oping agency,  determined  engineering  and  sci- 
entific factors.  Service  tests,  conducted  by  the 
using  agency,  determined  suitability  for  field 
use.  The  Secretary  of  War  usually  accepted  or 
rejected  the  item  based  on  the  user’s  evaluation. 

In  1940,  with  the  establishment  of  a re- 
search and  development  section  in  logistics, 
each  arm  and  service  performed  its  own  test 
and  evaluation.  Service  tests  were  assigned  to  a 
board  closely  connected  with  the  respective  ser- 
vice school. 

Under  direct  control  of  its  respective  arm 
of  the  service,  each  board  was  concerned  only 
with  equipment  used  by  that  arm.  A central  au- 
thority did  not  exist,  and  the  evaluation  by  one 
board  was  not  binding  on  the  others. 

In  World  War  II,  testing  was  accelerated 
and  six  new  boards  were  established.  At  one 


point,  because  of  heightened  demands  for 
equipment,  service  tests  were  conducted  during 
production,  and  modifications  were  incorpo- 
rated on  the  assembly  line.  The  New  Develop- 
ment Division,  established  in  1943  to  supervise 
testing  of  new  weapons  and  equipment,  sent 
teams  overseas  to  introduce  equipment  and  ob- 
serve equipment  performance  in  battle.  Theater 
commanders  still  made  the  final  decision,  how- 
ever. 

During  the  1950’s,  new  equipment  became 
obsolete  almost  as  soon  as  it  became  standard. 
It  was  obvious  that  pre-World  War  II  philoso- 
phy and  procedures  would  have  to  be  adapted 
to  the  rampant  technology  of  the  postwar  world 
or  the  United  States  would  fall  behind  in  mod- 
ernization. 

In  1962  the  Department  of  the  Army  acti- 
vated the  US  Army  Materiel  Command  (AMC) 
for  weapons  and  equipment  development  and 
procurement.  The  Army  Materiel  Command 
assigned  testing  responsibilities  to  the  Com- 
manding General,  US  Army  Test  and  Evalua- 
tion Command  (TECOM).  This  latter  Com- 
mand assumed  responsibility  for  the  test  and 
evaluation  missions  formerly  assigned  to  the 
technical  services. 

As  the  Army’s  principal  materiel  testing 
organization.  Test  and  Evaluation  Command 
was  assigned  the  basic  mission  of  providing  de- 
cisionmakers with  unbiased  independent  ap- 
praisals of  Army  materiel.  The  Army  chartered 
the  Test  and  Evaluation  Command  to  reduce 
the  time  frame  between  design  and  production, 
and  eliminate  duplication  of  effort  through  in- 
tegrated testing  and  better  coordination. 

To  accomplish  this  charter,  Test  and  Eval- 
uation Command  structured  its  operation 
around  two  traditional  test  types:  engineering 
and  service.  The  engineering  tests  revealed 
technical  flaws  in  design  that  should  be  elimina- 
ted by  the  producer.  The  Army  Test  and  Evalu- 
ation Command  then  placed  the  materiel  in  the 
hands  of  military  personnel  for  service  testing 
under  realistic  operational  conditions.  These 
tests  determined  the  equipment’s  suitability  for 
Army  use  and  release  for  production.  During 
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early  production,  initial  production  tests  deter- 
mined if  deficiencies  found  during  previous 
testing  were  corrected.  This  cleared  the  way  for 
full  production  of  the  item. 

In  1970,  the  Presidential  Blue  Ribbon  De- 
fense Panel  recommended  that  an  operational 
assessment,  independent  of  the  materiel  devel- 
oper, be  conducted  prior  to  a major  production 
purchase.  Because  of  Test  and  Evaluation 
Command’s  affiliation  with  Army  Materiel 
Command  developers,  criticism  arose  when  the 
Test  and  Evaluation  Command  expanded  its 
service  test  to  include  an  operational  exercise. 

As  a result,  the  Operational  Test  and  Eval- 
uation Agency  (OTEA)  was  established  (1972). 
The  Operational  Test  and  Evaluation  Agency 
was  given  the  management  responsibility  for  all 
operational  testing.  The  Operational  Test  and 
Evaluation  Agency  shares  the  management  re- 
sponsibility with  the  US  Army  Training  and 
Doctrine  Command  (TRADOC)  which  was 
formed  less  than  a year  later.  The  Army  Test 
and  Evaluation  Command  continued  to  per- 
form engineering  testing  and  provided  services 
for  the  operational  testers  because  much  of  the 
capability  for  operational  testing  remained  at 
Test  and  Evaluation  Command  test  boards. 

In  May  1973,  a joint  Army  Materiel  Com- 
mand/Industry meeting  was  held  in  Atlanta, 
Georgia.  The  outcome  of  this  conference  was  a 
recommendation  to  simplify  test  procedures 
and  eliminate  duplication  of  effort. 

This  recommendation  was  followed  by  a 
study  released  by  the  Army  Materiel  Acquisi- 
tion Review  Committee  in  October  1974.  This 
study  boosted  the  trend  toward  simplification 
of  the  test  process  through  mutual  exchange  of 
test  data.  Additionally,  the  Committee  study 
recommended  that  the  Test  and  Evaluation 
Command  test  boards  be  transferred  to  Train- 
ing and  Doctrine  Command  for  operational 
testing,  force  development  testing,  and  experi- 
mentation. The  change  drew  a greater  distinc- 
tion between  development  and  operational  test- 
ing— the  latter  superseding  service  testing. 

On  January  23,  1976,  Army  Materiel 
Command  changed  its  name  to  the  US  Army 


Materiel  Development  and  Readiness  Com- 
mand (DARCOM)  to  better  convey  its  two- 
fold mission  of  developing  materiel  and  making 
certain  the  materiel  is  ready  for  issue.  The  US 
Army  Materiel  Development  and  Readiness 
Command  has  a clear  responsibility  to  develop 
hardware  according  to  field  unit  needs,  and  to 
be  sure  the  hardware  is  supportable  in  the  field. 

The  evolution  of  Army  materiel  testing 
continues:  changing,  refining,  constantly  im- 
proving. The  idea  is  to  benefit  from  test  trials 
and  failures.  In  Army  testing,  the  past  is  cer- 
tainly a prologue  to  a successful  future. 

PRESENT  DEPARTMENT  OF 
ARMY  TEST  COMMUNITY 

Today,  Army  tests  are  of  two  categories: 
development  and  operational.  These  tests  have 
been  known  by  other  terms  but  still  retain  the 
traditional  meanings.  Development  tests,  per- 
formed by  scientists  and  technicians  or  spe- 
cially trained  soldiers  in  controlled  settings,  de- 
termine how  well  the  item  works.  Operational 
tests,  accomplished  by  the  ultimate  user  of  the 
item,  determine  if  a soldier  can  apply  the  item 
in  a tactical  environment. 

Development  testing  v&  conducted  to  dem- 
onstrate that  the  engineering  design  and  devel- 
opment process  is  complete;  that  design  risks 
have  been  minimized;  that  the  test  item  will 
meet  specifications;  and  to  estimate  the  sys- 
tem’s military  utility  if  made  part  of  the  inven- 
tory. This  testing  is  accomplished  in  the  fac- 
tory, laboratory  and  proving  ground. 

The  US  Army  Materiel  Development  and 
Readiness  Command  is  responsible  for  most  de- 
velopment testing  and  assigns  the  bulk  of  it  to 
the  Test  and  Evaluation  Command.  Histo- 
rically, Test  and  Evaluation  Command  has  pro- 
vided Army  decisionmakers  with  unbiased, 
independent  testing  and  reliable  test  data.  Now 
Test  and  Evaluation  Command  continues  to 
make  a valuable  contribution  in  the  research, 
development,  test  and  evaluation  effort,  but  as- 
sumes a markedly  different  posture.  Emphasis 
has  shifted  from  independent  testing  to  inde- 
pendent evaluation.  Test  data  is  collected  from 
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contractors  and  proponent  developers,  as  well 
as  from  tests  executed  by  the  Test  and  Evalua- 
tion Command,  to  support  the  evaluation. 

The  other  test  category,  operational  test- 
ing, is  conducted  to  estimate  the  prospective 
system’s  military  utility,  operational  effective- 
ness and  suitability,  and  need  for  modifications. 
Operational  testing  also  provides  information 
on  organization,  personnel  requirements,  doc- 
trine and  tactics  as  well  as  verification  of  asso- 
ciated operating  instructions,  publications  and 
handbooks.  Operational  testing  is  conducted  in 
as  realistic  an  operational  environment  as  possi- 
ble by  the  Operational  Test  and  Evaluation 
Agency  and  Training  and  Doctrine  Command. 
Both  answer  directly  to  the  Army  Chief  of 
Staff. 

The  Operational  Test  and  Evaluation 
Agency,  with  about  1 20  officers,  is  responsible 
for  test  design  of  major  and  designated  nonma- 
jor systems  and  provides  an  independent  evalu- 
ation. The  Infantry  Fighting  Vehicle  is  an  ex- 
ample of  a major  system.  The  XM71 1 High  Ex- 
plosive Projectile  for  the  8-inch  howitzer  is  an 
example  of  a nonmajor  system.  The  Opera- 
tional Test  and  Evaluation  Agency  either  ap- 
points a test  director  or  tasks  Training  and 
EKxtrine  Command  to  perform  the  test.  The 
US  Army  Forces  Command  (FORSCOM), 
along  with  the  Materiel  Development  and 
Readiness  Command  and  the  Training  and 
Doctrine  Command,  generally  provides  the 
military  organization  to  perform  the  test. 

The  Training  and  Doctrine  Command  is 
responsible  for  individual  training,  education 
and  combat  developments.  The  Training  and 
Doctrine  Command  also  conducts  operational 
tests  of  all  assigned  systems  and  provides  test 
design  and  evaluation  as  directed  for  nonmajor 
systems. 

To  conduct  tests.  Training  and  Doctrine 
Command  uses  subordinate  organizations. 
These  include  the  Training  and  Doctrine  Com- 
mand Arms  Test  Activity,  Fort  Hood,  Texas; 
the  Combat  Development  Experimentation 
Command,  Fort  Ord,  California;  and  seven  test 


boards.  The  boards  are  the  Airborne,  Commu- 
nications and  Electronics  Board,  Fort  Bragg, 
North  Carolina;  the  Armor  and  Engineer 
Board,  Fort  Knox,  Kentucky;  the  Air  Defense 
Board,  Fort  Bliss,  Texas;  the  Field  Artillery 
Board,  Fort  Sill,  Oklahoma;  the  Aviation  Test 
Board,  Fort  Rucker,  Alabama;  the  Intelligence 
and  Security  Board,  Fort  Huachuca,  Arizona; 
and  the  Infantry  Board,  Fort  Benning,  Georgia. 
In  addition.  Training  and  Doctrine  Command 
is  responsible  for  Force  Development  Testing 
and  Evaluation,  which  is  user  testing  to  deter- 
mine user  equipment  requirements. 


TEST  AND  EVALUATION  COMMAND 

The  Army  Test  and  Evaluation  Com- 
mand, with  headquarters  at  Aberdeen  Proving 
Ground,  Maryland,  directs  and  manages  nine 
test  agencies  within  the  continental  United 
States,  Alaska  and  Panama.  The  command 
controls  about  4 million  acres  of  real  estate. 
Acquisition  cost  of  Test  and  Evaluation  Com- 
mand installations — with  20-million  square  feet 
of  building  floor  space,  utilities  and  equip- 
ment— approaches  one-half  billion  dollars. 

Authorized  strength  is  almost  12,000,  in- 
cluding nearly  9,000  civilians.  Field  organiza- 
tions are  categorized  as  proving  grounds,  envi- 
ronmental centers,  a national  range  and  the 
Aircraft  Development  Test  Activity. 

Direct  management  of  test  operations  is 
done  in  Headquarters,  Test  and  Evaluation 
Command,  by  six  materiel  test  directorates 
(MTD)  and  the  Test  Operations  and  Policy  Of- 
fice. The  latter  monitors  and  coordinates  the  ef- 
forts of  the  materiel  test  directorates,  records 
status  of  all  test  efforts,  supervises  workloading 
of  subordinate  agencies  and  develops  timely  test 
policy.  The  materiel  test  directorates  have  over- 
all staff  responsibility  for  efficient,  economical 
and  effective  testing  of  materiel  and  are  orga- 
nized so  that  a single  major  system  usually  can 
be  handled  by  one  directorate  as  opposed  to  be- 
ing fragmented  among  many. 

In  addition  to  the  materiel  test  director- 
ates, five  other  directorates  address  systems 


10 


Defense  Systems  Management  Review 


r 


■rw 


analysis  and  evaluation,  test  design  and  statisti- 
cal analysis,  methodology,  human  factors  engi- 
neering and  instrumentation,  among  other 
considerations. 

The  Army  Test  and  Evaluation  Com- 
mand’s installations  and  activities  look  like  this; 

Aberdeen  Proving  Ground,  Maryland,  is 
the  Army’s  oldest  proving  ground.  All  weap- 
ons, except  nuclear  explosives  and  long-range 
missiles,  can  be  tested  there.  Instrumented  fir- 
ing of  weapons  up  to  ranges  of  42,000  yards  is 
possible  for  the  100  firing  positions  available. 
The  post  and  its  waters  occupy  82,000  acres.  A 
chief  facility  is  the  Munson  Test  Area  contain- 
ing what  has  been  called  the  “World’s  Worst 
Road”  on  which  wheeled  and  tracked  vehicles 
are  tested. 

Cold  Regions  Test  Center,  Fort  Greely, 
Alaska,  occupies  more  than  750,000  acres  of 
typical,  arctic  terrain  and  is  located  about  180 
miles  below  the  Arctic  Circle.  Firing  ranges,  ve- 
hicular test  courses,  improved  terrain  areas, 
and  maintenance  shops  are  used  to  perform  en- 
vironmental tests  on  all  types  of  materiel  under 
extreme  natural  conditions. 

The  US  Army  Aircraft  Development  Test 
Activity  is  a tenant  activity  at  Fort  Rucker,  Al- 
abama. The  Aircraft  Development  Test  Activ- 
ity has  the  responsibility  to  plan,  conduct,  eval- 
uate and  report  on  government  test  elements 
and  to  monitor,  evaluate  and  report  on  contrac- 
tor test  elements  of  aircraft  components  and 
aircraft-related  support  equipment. 

The  US  Army  Dugway  Proving  Ground, 
Utah,  conducts  field  and  laboratory  tests  to 
evaluate  chemical  and  radiological  defense  sys- 
tems and  materiel,  as  well  as  biological  defense 
research.  Dugway  Proving  Ground  serves  as 
the  DOD  joint  control  point  for  chemical  and 
biological  defense  data.  At  this  installation, 
840,000  acres  of  Great  Salt  Lake  Desert  are 
used  in  conducting  investigations  in  biological 
research  and  chemical,  radiological,  meteoro- 
logical, ecological  and  epidemiological  areas. 


The  Electronic  Proving  Ground,  Fort 
Huacbuca,  Arizona,  tests  communications  and 
electronics  equipment  and  systems  for  the  mili- 
tary services.  The  proving  ground  is  situated 
between  Yuma  Proving  Ground  and  White 
Sands  Missile  Range  and  provides  overlapping, 
compatible  instrumentation  facilities  for  several 
types  of  inflight  test  programs.  The  clear  elec- 
tromagnetic environment,  the  excellent  cli- 
matic conditions,  and  the  freedom  from  aircraft 
congestion  make  this  an  unusually  fine  area  for 
electronics  testing. 

Jefferson  Proving  Ground,  Indiana,  is  de- 
voted to  testing  ammunition.  This  is  the  Army’s 
smallest  Test  and  Evaluation  Command  instal- 
lation, occupying  55,000  acres  of  southeastern 
Indiana.  The  proving  ground’s  firing  ranges  can 
accommodate  all  weapons  and  weapons  sys- 
tems from  grenade  launchers  and  small  arms  to 
240-mm  howitzers. 

White  Sands  Missile  Range,  Sew  Mexico, 
with  4,000  square  miles,  is  the  country’s  largest 
military  reservation.  On  July  16,  1945,  the  first 
atomic  explosion  was  triggered  at  Trinity  Site 
in  a remote  area  of  the  1-week  old  post.  The 
range  has  gained  fame  for  testing  the  moon  lan- 
der, setting  world  records  for  reception  of 
highly  resolved  photographs  from  weather  sat- 
ellites, and  for  diverse  operations  from  nurtur- 
ing exotic  wild  game  to  using  the  nation’s  larg- 
est solar  furnace  that  generates  up  to  5,000  de- 
grees Fahrenheit  in  environmental  experi- 
ments. In  addition,  the  White  Sands  Missile 
Range  conducts  tests  for  the  Army,  Air  Force, 
Navy,  National  Aeronautics  and  Space  Admin- 
istration, Department  of  Energy  and  other  fed- 
eral agencies.  The  mission  is  to  assess  the  per- 
formance of  missiles,  missile  systems,  space 
communications,  test  methodology  and  radia- 
tion propagation  devices.  A chief  asset  of  the 
range  is  its  extremely  precise  instrumentation. 
Instrumentation  here  includes  telemetry,  pho- 
togrammetry,  real  time  data  networks,  trans- 
mission and  communications  associated  with 
test  and  evaluation. 

Tropic  Test  Center,  Panama  Canal  Zone, 
is  suited  for  tropical  environment  testing  of  ma- 
teriel. Located  600  miles  north  of  the  equator. 
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this  test  area  has  several  facilities  that  provide  a 
large  variety  of  landforms,  vegetation,  soil,  top- 
ographic, hydrological  and  climatic  conditions 
for  test  and  research  projects. 

Yuma  Proving  Ground,  Arizona,  occupies 
nearly  a million  acres  in  the  Sonoran  Desert. 
The  proving  ground’s  initial  mission  was  desert 
environment  testing.  Over  the  years  the  mission 
and  function  of  Yuma  Proving  Ground  have 
grown  owing  to  post  size  and  the  relative  free- 
dom from  adverse  weather  interference.  Now, 
like  Aberdeen  Proving  Ground,  test  and  evalu- 
ation are  conducted  at  Yuma  Proving  Ground 
in  a wide  range  of  areas  including  weapons, 
munitions  and  automotive  items.  The  Yuma 
Proving  Ground  has  a multipurpose  space  posi- 
tion range  designed  to  meet  the  need  of  aircraft 
armament  tests.  This  installation  also  has  more 
than  a dozen  vehicular  test  courses  for  automo- 
tive testing  including  sand  dunes,  rock  and 
gravel  courses,  and  water  basins  for  amphibious 
vehicles. 

CURRENT  METHOD  OF 
MATERIEL  ACQUISITION 

Today,  the  Army  has  four  basic  methods 
to  use  in  filling  its  materiel  needs.  The  Army 
can  purchase  existing  commercial  or  foreign  de- 
veloped products.  This  provides  a low  cost, 
quick  response  to  certain  requirements.  Second, 
the  Army  can  modify  commercial  or  foreign  de- 
veloped items.  This  method  also  provides  for  a 
quick  response  and  reasonably  low  cost.  Third, 
the  Army  can  choose  to  product  improve  al- 
ready existing  US  military  equipment  to  meet 
new  requirements.  The  fourth  method  is  to  ini- 
tiate a new  development  program.  The  advan- 
tages in  the  last  option  are  that  the  new  item 
can  meet  exact  military  requirements:  the  de- 
sign and  configuration  will  be  Government 
controlled  and  logistic  supportability  will  be 
enhanced. 

No  matter  which  of  the  four  options  is 
chosen,  the  items  must  be  tested  to  insure  that 
they  will  work  under  realistic  combat  condi- 
tions. For  the  purpose  of  this  discussion,  the 
fourth  option  or  a new  development  program 
has  been  selected. 


The  Army  process  for  developing  and 
fielding  new  items  of  equipment  is  formalized  in 
a management  model  called  the  Life  Cycle  Sys- 
tem Management  Model  (LCSMM).  The  Life 
Cycle  System  Management  Model  is  divided 
into  four  phases.  The  phases  are  the  program 
initiation  phase,  the  validation  and  demonstra- 
tion phase,  the  full-scale  engineering  develop- 
ment phase  and  the  production  and  deployment 
phase. 

As  an  item  moves  from  one  stage  to  the 
next,  prototypes  become  better  defined.  Each 
phase  includes  both  development  and  op- 
erational tests.  These  tests  are  conducted  as 
early  as  possible  and  throughout  the  materiel 
acquisition  process  to  reduce  acquisition  risks 
and  to  assess  military  worth.  Schedules  provide 
for  accomplishing  test  and  evaluation  mile- 
stones prior  to  key  decision  points. 

The  latest  approach  to  Army  development 
testing  is  the  Single  Integrated  Development 
Test  Cycle  (SIDTC).  The  Single  Integrate  De- 
velopment Test  Cycle  aims  at  a truly  integrated 
development  test  with  everybody  on  board  at 
the  start — the  developer,  contractor,  tester, 
evaluator  and  logistician.  The  Test  Cycle  em- 
bodies the  objectives  of  reducing  costs  and  sav- 
ing development  time  and  hardware  by  combin- 
ing test  phases  and  using  contractor  test  data. 

Before  the  Single  Integrated  Development 
Test  Cycle,  the  Army’s  test  cycle  was  almost 
entirely  sequential.  There  was  duplication  and 
room  for  criticism.  Testing  was  redundant  and, 
in  some  instances,  not  particularly  cost  effec- 
tive. 

The  primary  objective  of  the  Single  Inte- 
grated Development  Test  Cycle  is  to  eliminate 
duplication  in  development  testing.  The  SIDTC 
emphasizes  the  total  integration  of  develop- 
ment testing  and  calls  for  consideration  of  com- 
bining, and  the  concurrency  of,  some  phases  of 
development  and  operational  testing.  Impor- 
tant contributions  have  been  the  introduction 
of  greater  precision  and  discipline  into  testing 
and  the  emphasis  placed  on  independence  of 
evaluation.  For  SIDTC  to  work,  the  entire  test 
and  evaluation  community  needs  to  coordinate 
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its  requirements  and  programs  as  early  as  possi- 
ble in  the  development  cycle. 

There  are  five  blocks  for  implementing  the 
Single  Integrated  Development  Test  Cycle.  The 
blocks  represent:  coordination  of  effort,  utiliza- 
tion of  contractor  test  data,  integration  of  tests, 
increased  early  soldier  participation,  and  em- 
phasis on  independence  of  evaluation. 

Consider  coordination  of  effort.  Program 
management  responsibility  is  now  focused 
more  than  ever  on  the  materiel  developer.  The 
materiel  developer  must  coordinate  the  efforts 
of  all  participants  and  integrate  valid  require- 
ments into  cost  effective  development.  One  of 
the  principal  means  of  accomplishing  this  coor- 
dination is  for  the  developer — actually  the 
project  manager — to  form  and  chair  test  inte- 
gration working  groups.  The  developer,  con- 
tractor, development  and  operational  testers 
and  evaluators,  and  the  logistic  support  people 
are  intimately  involved  in  this  coordination 
process.  This  group  must  be  composed  of  coop- 
erative individuals  with  the  authorization  to 
make  concessions  with  respect  to  the  parochial 
interests  of  the  participants.  The  materiel  devel- 
oper and  his  test  integration  working  group  es- 
tablish and  review  the  Coordinated  Test  Pro- 
gram to  insure  that  maximum  integration  has 
been  effected  and  that  the  test  design  is  appro- 
priate and  cost  effective. 

The  SIDTC  concept  requires  maximum 
utilization  of  contractor  test  results.  In  the  past, 
the  Army  received  and  utilized  contractor  data 
and  reviewed  contractor  test  plans  to  a limited 
extent.  Now  the  Army  is  becoming  more  in- 
volved in  monitoring  contractor  testing  and  us- 
ing contractor  test  data.  Army  access  to  con- 
tractor data  is  steadily  increasing  as  the  Army 
is  kept  better  informed  about  current  and  pro- 
posed contractor  test  planning. 

The  contractor’s  share  of  testing  is  being 
expanded,  and  test  resources  at  contractor  dis- 
posal are  being  increased.  In  effect,  the  Army’s 
development  test  capability  at  proving  grounds 
and  ranges  at  major  Army  installations  is  avail- 
able to  the  contractor,  with  reimbursement 


based  on  direct  costs  only.  When  feasible,  in- 
dustry can  conduct  contractor  tests  at  Govern- 
ment test  sites  with  contractor  personnel.  Gov- 
ernment personnel  can  also  perform  tests  on  a 
customer  basis. 

The  reduction  of  testing  time  and  the  inte- 
gration of  developmental  effort  are  basic  objec- 
tives of  SIDTC.  The  Army  has  often  been  criti- 
cized for  taking  too  much  time  in  testing.  Inte- 
gration of  test  phases  is  one  way  to  achieve 
production  decision  points  earlier.  Concurrent 
testing  between  development  and  operational 
testing  is  both  feasible  and  practicable.  In  the 
past,  operational  testing  often  appeared  as  a 
separate  block  of  time  at  the  completion  of  de- 
velopment testing.  Now,  the  Army  must  be  pre- 
pared to  realign  its  test  programs  to  allow  for 
concurrency,  when  concurrency  is  feasible,  de- 
sirable and  makes  economic  sense. 

During  the  materiel  acquisition  process, 
items  are  taken  through  three  stages  of  develop- 
ment and  operational  testing.  The  first  de- 
velopment test  (DT  I)  is  conducted  early  during 
the  validation  and  demonstration  phase  to  dem- 
onstrate, fundamentally,  that  technical  risks 
have  been  identified  and  that  solutions  are  in 
hand.  The  first  operational  test  (OT  I)  is  con- 
ducted during  the  validation  phase  to  provide 
an  indication  of  military  utility  and  worth  to 
the  user. 

Development  test  II  (DT  II)  tests  an  engi- 
neering development  prototype  during  the  full- 
scale  engineering  development  phase.  The  DT 
II  provides  final  technical  data  for  determining 
a system’s  readiness  for  transition  into 
production.  Operational  test  II  (OT  II),  con- 
ducted in  as  realistic  an  operational  environ- 
ment as  possible,  tests  issues  best  examined  by 
troop  units  in  controlled  field  exercises. 

Development  test  III  (DT  III)  is  con- 
ducted on  limited  production  items  to  verify  the 
adequacy  and  quality  of  materiel  when  manu- 
factured. The  Development  Test  III  is  not  re- 
quired if  Development  Test  II  is  successfully 
completed  and  the  system  progresses  directly  to 
full  production.  Operational  Test  III  tests  ini- 
tial production  items  to  verify  that  all  critical 
issues  have  been  resolved.  When  a full-scale 
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production  decision  is  made,  the  subsequent 
testing  may  include  production  testing  and,  if 
necessary,  follow-on  evaluation  testing. 

The  Army  must  consider  not  only  the  inte- 
gration of  Government  and  contractor  testing, 
but  also  the  integration  of  Development 
Test/Operational  Test,  especially  in  Develop- 
ment Test  I/Operational  Test  I and  Develop- 
ment Test  Ill/Operational  Test  III  phases  of 
the  life  cycle.  With  respect  to  Development 
Tests/Operational  Tests  I and  III,  the  Army 
must  consider  also  the  possibility  of  a reduction 
in  scope  of  these  programs.  This  may  be  possi- 
ble if  the  scope  of  the  Development  Test  II/Op- 
erational  Test  II  effort  is  more  comprehensive. 
The  Operational  Test  and  Evaluation  Agency  is 
currently  investigating  how  to  integrate  its  re- 
quirements for  Operational  Test  I and  Opera- 
tional Test  III  with  appropriate  Development 
Test  phases. 

Too  much  modification  of  requirements 
could  be  costly.  Reliability,  that  driving  factor 
behind  the  duration  of  most  development  test- 
ing, may  come  to  haunt  us  in  the  future.  Experi- 
ence has  shown  that  reliability  values  reported 
from  the  field  are  usually  lower  than  those  pre- 
dicted by  design  analysis,  or  demonstrated  by 
laboratory  testing.  Since  all  participants  are 
concerned  about  reliability,  a new  Department 
of  Defense  approach  may  help.  Efforts  are  un- 
derway to  establish  a uniform  set  of  reliability 
terms  and  definitions  that  can  be  tracked 
through  all  phases  of  research,  development, 
test  and  evaluation.  When  this  is  done,  the 
Army/Industry  team  will  have  an  improved 
basis  upon  which  to  evaluate  reliability. 

Early  user  or  troop  participation  in  the  de- 
velopment cycle,  is  the  fourth  major  element. 
This  emphasis  has  already  injected  troop  know- 
how into  some  of  our  top  priority  weapon  sys- 
tems. Solutions  to  user  equipment  problems, 
surfaced  early  in  the  development  cycle,  insure 
compatibility  of  critical  man-machine  relations 
and  acceptability  of  the  design  strategy. 

Finally,  the  new  emphasis  on  independent 
evaluation,  with  respect  to  the  developer,  is  re- 
sulting in  more  efficient  test  designs  and  pro- 
gram structures.  Additionally,  it  increases  the 


credibility  of  the  process.  The  increased  techni- 
cal planning  requires  intensive  early  manage- 
ment. It  is  expected  that  this  early  commitment 
of  increased  resources  will  generate  substantial 
savings  as  the  program  develops. 

The  Single  Integrated  Development  Test 
Cycle  has  been  in  full  gear  for  2 years.  The  me- 
thod is  continually  being  fine-tuned  to  be  more 
responsive  to  the  developer  and  decisionmak- 
ers. Inroads  are  being  made  into  reducing  costs 
and  the  length  of  the  equipment  acquisition  cy- 
cle. 

It  is  difficult  to  draw  a comparison  be- 
tween the  efficiency  of  testing  as  performed 
prior  to  SIDTC  and  testing  as  performed  now. 
Systems  currently  under  evaluation  were 
largely  developed  prior  to  1975.  However,  the 
primary  objective  of  SIDTC  was  to  eliminate 
unnecessary  duplication  in  development  test- 
ing. That  has  been  accomplished  to  a large  ex- 
tent. The  total  integration  of  development  test- 
ing and  the  combining  of  some  phases  of 
Development  Test  and  Operational  Test  have 
received  an  encouraging  start. 

The  most  important  contribution  of  the 
Single  Integrated  Development  Test  Cycle  has 
been  to  introduce  more  precision  and  discipline 
into  testing  and  to  emphasize  the  independence 
of  evaluation.  Critical  test  issues  are  defined  so 
that  Development  Test  and  Operational  Test 
can  be  evaluated  on  a valid  basis.  Sequential  de- 
velopment testing  has  been  significantly  re- 
duced. Test  integration  working  groups  are  co- 
ordinating the  efforts  of  all  participants  and  in- 
tegrating valid  requirements  into  cost  effective 
development. 

Despite  problems  arising  from  the  intro- 
duction of  the  new  system,  large  dollar  and  ma- 
teriel savings  have  been  identified  in  specific  de- 
velopment programs.  Continuing  refinements 
point  to  further  gains. 

For  example,  in  the  Ground  Emplaced 
Mine  Scattering  Systems  program,  three  test  in- 
tegration working  group  meetings  were  held  to 
review  contractor  engineer  design  and  Test  and 
Evaluation  Command  test  programs.  Test  item 
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reductions  of  approximately  80  percent  were 
accomplished  through  the  integration  of  devel- 
opment test,  operational  test,  contractor  and 
developer  tests.  Some  redujidant  subtests  were 
completely  eliminated.  The  need  for  more  than 
1,000  mines  was  deleted  from  Test  and  Evalua- 
tion Command  development  test  phases.  Fur- 
ther reductions  were  achieved  as  a result  of 
changes  to  testing  methods  and  procedures, 
and  the  use  of  dummy,  less  costly  prototype 
mines. 

The  Stinger,  which  is  a shoulder-fired,  air 
defense  weapon  system,  is  the  planned  replace- 
ment for  the  older  Redeye.  Significant  savings 
in  the  development  testing  program  were 
achieved  by  test  integration  working  group  ef- 
forts. In  terms  of  development  time.  Stinger 
should  be  fielded  in  8 years,  as  compared  to  1 3 
years  for  Redeye.  While  approximately  500 
missiles  were  fired  in  developing  and  testing  the 
Redeye,  only  130  missiles  were  fired  to  develop 
and  test  the  Stinger.  The  Redeye  experienced 
cost  overruns  of  about  300  percent  in  research, 
development,  test  and  evaluation  costs.  The 
Stinger  had  only  a 15  percent  cost  overrun  in 
the  same  areas. 

Another  example  of  SIDTC  application 
occurred  in  a new  area  of  weapons  develop- 
ment. The  Roland  surface  to  air  missile,  a Euro- 
pean weapon  system,  was  adapted  for  US  test- 
ing and  production.  Here  the  emphasis  was  on 
increased  cooperative  testing  with  the  French 
and  Germans  at  European  and  US  test  sites. 
The  thrust  was  to  obtain  more  usable  data  per 
missile  from  European  tests  in  lieu  of  tests  to  be 
conducted  later  in  the  United  States.  The  d^ta 
is  to  be  obtained  by  using  US  telemetry  on  the 
European  missiles  and  providing  US  high- 
performance  aircraft  targets. 

The  Army  was  interested  also  in  earlier 
operational  tester  participation,  testing  of  sup- 
port equipment  and  initiation  of  training.  Con- 
versely, the  Europeans  were  more  concerned 
with  the  interchangeability  of  parts,  interopera- 
bility of  subsystems  and  design  control  of  the 
US  and  European  versions  of  the  system. 

A test  review  team,  including  US  contrac- 
tors, devised  a cooperative  test  program  that 


was  successfully  negotiated  in  principle  with 
the  partner  countries.  The  price  the  Army  paid 
in  this  case  included  additional  front-end  costs, 
but  no  increase  in  the  overall  program.  A for- 
mal set  of  agreements  with  the  Europeans  com- 
mits the  United  States  to  detailed  procedures  of 
design  control  and  adherence  to  the  extensive 
testing  support  offered  by  the  United  States. 

The  United  States  learned  much  in  this 
reverse  flow  of  weapon  systems  technology 
from  Europe.  There  is  a difference  in  test  phi- 
losophy, procedures,  standards  and  techniques. 
If  this  sharing  is  to  continue  and  progress,  the 
United  States  must  be  prepared  for  the  ramifi- 
cations of  technology  transfer  and  the  impact  of 
the  metric  system  on  drawings,  tools  and  in- 
strumentation. 


INTERNATIONAL 
MATERIEL  EVALUATION 

The  United  States  is  interested  in  sharing 
development  with  European  countries.  In  No- 
vember 1976,  as  a result  of  Presidential,  Con- 
gressional and  Department  of  Defense  concern 
that  the  Army  was  not  doing  enough  in  this 
area,  a new  program  was  established.  The 
Army  Test  and  Evaluation  Command  was 
given  the  responsibility  for  evaluating  foreign 
military  weapons  and  equipment  for  possible 
US  procurement  as  alternatives  to  developmen- 
tal items. 

The  new  mission  will  be  accomplished 
with  a small  staff  element  at  Test  and  Evalua- 
tion Command  headquarters.  The  Test  Opera- 
tions and  Policy  Office  will  serve  as  the  point  of 
contact. 

The  new  program  will  involve  evaluating 
foreign  systems  that  meet  US  inventory  re- 
quirements. The  object  in  considering  foreign 
systems  is  to  obtain  improved  capability,  de- 
creased costs,  earlier  operational  availability 
and  an  optimum  degree  of  NATO  standardiza- 
tion and  interoperability,  within  the  constraints 
of  existing  US  laws  and  regulations.  The  evalu- 
ation process  is  divided  into  three  phases. 
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Phase  one  is  the  identification  of  possible 
foreign  systems  for  evaluation.  After  a require- 
ment is  made  known,  a foreign  capability 
search  will  be  made.  If  potential  condidates  are 
identified,  Test  and  Evaluation  Command  will 
attempt  to  verify  the  information  that  justified 
the  appearance  of  the  name  of  a foreign 
system(s)  on  the  candidate  list.  An  inprocess  re- 
view committee  is  to  decide  which  items  merit  a 
further,  indepth  evaluation. 

In  the  second  phase,  more  foreign  data 
gathering  is  necessary.  Evaluations  to  this  point 
usually  are  made  on  data  existing  within  US 
agencies.  If  the  item  is  in  production  or  de- 
ployed, copies  of  foreign  development  test  re- 
ports will  be  requested.  The  Test  and  Evalua- 
tion Command  will  draft  an  evaluation  plan  to 
analyze  and  validate  the  data  gathered.  From 
this,  a preliminary  evaluation  is  to  be  drafted. 
Again,  the  inprocess  review  committee  should 
decide  either  that  further  effort  will  be  ex- 
pended in  the  program  or  that  the  foreign  alter- 
native is  not  viable. 

In  the  third  phase,  a limited  test  program 
is  to  provide  the  information  not  available 
through  analysis  of  existing  data.  Funding  will 
be  established  and  the  test  integration  working 
group  process  should  determine  the  extent  of 
testing  to  provide  a final  evaluation.  At  the  end 
of  this  stage,  regardless  of  the  inprocess  review 
committee’s  decision,  the  Test  and  Evaluation 
Command  role  is  complete.  Further  effort  fol- 
lows normal  materiel  acquisition  guidelines. 

Currently,  Data  Exchange  Agreements  in 
testing  and  evaluation  exist  among  the  United 
States  and  some  foreign  countries.  The  new  In- 
ternational Materiel  Evaluation  Program  will 
utilize  these  agreements  during  the  first  phase 
of  the  program  to  increase  information  ex- 
change and  coordination. 

Most  of  this  exchange  now  is  accom- 
plished after  each  country  has  spent  consider- 
able funds  on  independent  research  and  devel- 
opment projects.  This  new  program  should 
help  eliminate  duplication  in  research  and  de- 
velopment areas.  Hopefully,  the  countries  will 
share  expertise  and  funds.  The  push  for  more 


integration  and  coordination  with  the  Army’s 
research  and  development  sector  thus  spreads 
into  development  areas  shared  with  foreign 
governments. 

DARCOM  TEST  FACILITIES  REGISTER 

With  the  implementation  of  the  Single  In- 
tegrated Development  Test  Cycle,  industry  has 
a much  greater  role  and  responsibility  in  devel- 
opment testing.  Industry  must  recognize  that 
adequate  test  facilities  are  an  essential  part  of 
cost  effective  testing.  This  is  not  to  say  that  in- 
dustry should  begin  developing  extensive 
ranges,  test  facilities  or  instrumentation,  but 
that  industry  should  make  maximum  use  of 
what  is  available,  especially  government  facili- 
ties. 

For  its  part  of  the  program.  Test  and  Eval- 
uation Command  is  taking  concerted  actions  to 
use  all  valid  data  from  contractors.  Further- 
more, we  are  counting  on  industry  for  im- 
proved test  data.  There  remain  situations  where 
contractor  test  facilities  are  inadequate.  This  of- 
ten results  in  data  that  can  not  be  used  and  ne- 
cessitates additional  costs  for  repeating  the  test 
at  a government  facility. 

The  use  of  Test  and  Evaluation  Command 
facilities  by  industry  is  not  new.  At  White 
Sands  Missile  Range,  Test  and  Evaluation 
Command  has  conducted  contractor  testing  for 
many  years.  Missile  firing  tests  conducted  by 
contractor  crews  are  a common  occurrence.  At 
Aberdeen  Proving  Ground,  Test  and  Evalua- 
tion Command  conducted  contractor  testing  of 
the  two  US  XMl  prototypes  to  the  full  satis- 
faction of  both  contractors. 

The  Single  Integrated  Development  Test 
Cycle  has  emphasized  and  formalized  the  proc- 
ess of  contractor  testing  at  government  facili- 
ties. A new  Army  Materiel  Development  and 
Readiness  Command  regulation,  70-61  (Apr 
77),  will  assist  contractors  in  using  government 
facilities.  The  new  regulation  defines  test  facili- 
ties as  an  inclusive  term  covering  real  estate, 
airspace,  buildings,  grounds,  instrumentation, 
vehicles,  equipment  and  personnel.  But,  the 
regulation  applies  only  to  those  cases  where  a 
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government  contract  is  involved.  Where  a gov-^ 
emment  contract  is  not  in  being,  a separate  reg- 
ulation covers  the  use  of  Test  and  Evaluation 
Command  test  facilities  by  private  industry. 

The  utilization  policy  expressed  in  the  new 
regulation  addresses  three  main  issues.  The  first 
states  Test  and  Evaluation  Command  test  facil- 
ities will  be  made  available  for  use  by  contrac- 
tors when  such  arrangements  prove  to  be  cost 
effective  to  the  government.  The  second  em- 
phasizes that  provisions  for  use  of  Test  and 
EvaUiation  Command  test  facilities  will  be  spe- 
cifically included  with  applicable  requests  for 
proposals,  invitations  for  bid  and  government 
contracts  under  negotiation.  The  third  issue 
stresses  that  priorities  for  test  facility  use  by 
contractors  will  be  equivalent  to  priorities  af- 
forded respective  Army  materiel  development 
and  production  programs. 

The  new  regulation  simplifies  and  defines 
the  requirements  that  contractors  must  follow 
to  use  Test  and  Evaluation  Command  facilities. 
The  regulation  should  benefit  both  industry 
and  the  US  Army.  However,  it  is  not  enough  to 
know  that  government  facilities  are  available  to 
contractors.  Industry  must  know  what  facilities 
exist. 

The  Test  and  Evaluation  Command  has 
prepared  a two-volume  DARCOM  Test  Facili- 
ties Register  to  ease  identification  and  selection 
of  test  capabilities.  Volume  I,  listing  29  Army 
Materiel  Development  and  Readiness  Com- 
mand installations  and  activities,  is  for  general 
reference  and  describes  major  facilities  worth 
$50,000  or  more.  Each  facility  is  described  as  to 
what  it  is  and  what  it  can  do.  Environmental 
features  and  constraints  also  appear. 

Volume  I was  published  in  May  1976.  The 
Test  and  Evaluation  Command  is  updating  that 
volume  to  include  annexes  that  describe  40 
other  Department  of  Defense  test  facilities  as 
well  as  60  contractor  facilities  that  have  pro- 
vided test  support  to  the  Army  during  the  last  5 
years. 

Volume  II  is  a computer  listing  of  instru- 
mentation and  equipment  (20,000  pieces  each 


worth  $3,(XX)  or  more)  designed  to  invite  com- 
parison of  various  options. 

For  the  first  time,  all  users  in  the  research 
and  development  community  now  have  avail- 
able a complete  statement  of  test  instru- 
mentation and  test  facilities.  The  Register  pro- 
vides a single-source,  clearinghouse  service.  By 
using  the  Register,  unnecessary  duplication  of 
facilities  with  the  Army  can  be  avoided.  Pro- 
gram dollars,  through  the  use  of  existing  re- 
sources, can  be  concentrated  in  hardware  devel- 
opment and  not  be  diverted  to  buy  already 
available  instrumentation. 

The  Register  will  permit  more  intelligent 
planning  and  programming  of  instrumentation, 
improvement  and  modernization  funds.  Short- 
falls in  resources  can  be  recognized  earlier,  be- 
fore test  facilities  are  needed  for  immediate 
project  support. 

Use  of  the  Register  should  facilitate  coop- 
eration between  Army  and  non-Army  testers — 
other  Department  of  Defense  agencies  and  in- 
dustry. The  Register  should  allow  for  cost  re- 
duction and  production  of  more  reliable  data. 
Even  when  development  testing  occurs  outside 
Test  and  Evaluation  Command,  that  command 
maintains  responsibility  for  test  execution. 
When  any  test  facility  is  used.  Test  and  Evalu- 
ation Command  must  be  assured  that  the  test- 
ing is  conducted  under  conditions  that  meet 
Army  requirements. 

OBSERVATIONS 

Today’s  testing  methods  are  more  effective 
than  those  of  20  years  ago.  As  weapon  systems 
and  equipment  become  more  sophisticated  and 
complex,  test  planning  and  procedures  become 
more  complicated. 

The  Single  Integrated  Development  Test 
Cycle,  with  its  thrust  for  test  integration  and 
Army/Industry  coordination,  has  been  in  op- 
eration for  2 years.  Improvements  in  the  sys*em 
are  measurable.  But,  there  has  been  a price  to 
pay  for  these  innovations.  Additional  costs 
were  often  incurred  early  and  management 
risks  were  increased.  Overall,  some  time  and 
money  savings  have  resulted. 
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The  Single  Integrated  Development  Test 
Cycle  is  not  perfect.  As  the  policy  is  used,  areas 
for  improvement  become  noticeable.  Industry 
must  provide  the  Army  with  reliable  data. 
More  usable  contractor  test  data  would  reduce 
government  testing  time  and  scope.  Theoreti- 
cally, if  all  requirements  in  the  development 
test  phase  could  be  met  by  the  contractor,  using 
contractor  or  Army  facilities,  some  of  the  ulti- 
mate objectives  of  the  Single  Integrated  Devel- 
opment Test  Cycle  would  be  guaranteed. 


In  the  final  analysis,  to  reduce  the  duplica- 
tion and  unnecessary  redundancy  still  in  the 
materiel  acquisition  cycle,  the  cooperation  be- 
tween the  Army  and  industry  must  improve. 
The  potential  to  solve  the  problems  associated 
with  growing  costs  and  rampant  technology 
does  exist.  Ever-increasing  efforts  to  pool  data, 
expertise  and  facilities  must  continue.  The 
Army  continues  to  change,  but  we  appear  to  be 
on  the  right  track  for  improved  testing  of  weap- 
ons systems. 


On  the  operational  testing  side,  hardware 
requirements  and  logistics  support  have  been 
strong  points,  but  training  and  personnel  sup- 
port areas  often  have  been  troublesome. 

As  weapon  system  complexity  increases, 
maintenance  becomes  more  difficult.  Also  it  is 
more  difficult  to  train  soldiers  to  use  the  com- 
plex systems.  Therefore,  at  the  front  end  of  the 
development  cycle,  user  requirements  must  be 
specific  so  training  packages  can  be  complete  to 
ready  soldiers  for  operational  testing.  By  the 
time  of  Operational  Test  I,  troops  should  be 
trained  to  use  the  equipment  so  that  the  United 
States  Army  Training  and  Doctrine  Command 
can  determine  how  well  the  system  measures  up 
to  desired  performance  levels.  In  that  way, 
trainers  know  what  kind  of  simulators  will  be 
necessary  for  training.  Furthermore,  contrac- 
tors know  earlier  if  the  equipment  must  be  sim- 
plified because  it  is  too  complex  for  the  skill 
level  of  the  soldier  who  is  going  to  operate  it. 
The  most  reliable  system  is  never  effective  if  a 
soldier  can  not  use  it  in  the  operational  envi- 
ronment. 

With  SIDTC,  the  time  allotted  for  devel- 
oping instrumentation  and  facilities  for  testing 
has  been  shortened.  Previously,  when  testing 
was  sequential,  new  instrumentation  develop- 
ment could  be  postponed.  Now,  with  several 
phases  starting  at  once,  instrumentation  must 
be  ready  from  the  outset.  Early  in  the  life  cycle, 
test  facilities  and  instrumentation  requirements 
must  be  identified.  Then,  with  the  use  of  the 
DARCOM  Test  Facilities  Register,  a determi- 
nation can  be  made  about  what  requirements 
must  be  developed. 
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IMPROVING  AIR  FORCE  INDEPENDENT 
OPERATIONAL  TESTING 


by 

Major  General  Howard  W.  Leaf 
Commander,  Air  Force  Test  and  Evaluation  Center 
Kirtland  Air  Force  Base,  New  Mexico 


The  Air  Force  Test  and  Evaluation  Center  manages  the  Air  Force’s  Operational  Test  and 
Evaluation  program  and  independently  evaluates  and  reports  on  the  operational  test  and 
evaluation  of  new  Air  Force  weapon  systems  in  compliance  with  Air  Force  Regulations  AFR 
23-36'  and  AFR  80-14/  The  Air  Force  Test  and  Evaluation  Center  was  established  on  1 
January  1975  to  fulfill  Department  of  Defense  and  Congressional  desires  that  each  service 
branch  have  an  agency  separate  and  distinct  from  the  developing  and  using  commands  to 
conduct  operational  testing. 

In  this  article  the  initiatives  taken  to  improve  operational  testing  are  addressed  and 
examples  of  the  steps  the  Air  Force  Test  and  Evaluation  Center  is  taking  to  put  these  initiatives 
into  practice  are  provided.  Before  examining  these  improvements  to  operational  test  and 
evaluation,  a brief  history  of  Air  Force  Test  and  Evaluation  Center  involvement  in  operational 
testing  is  appropriate. 

INTRODUCTION 

1 

Operational  test  and  evaluation  has  re- 
ceived so  much  attention  during  the  past  3 or  4 
years  that  it  is  sometimes  thought  to  be  a 
distinctively  new  concept.  It  is  not.  Within  the 
evolution  of  American  Air  Power  there  is  a 
long  history  of  both  formal  and  informal  test 
and  evaluation.  In  fact,  I would  suggest  that 
most,  if  not  all,  technological  improvements 
have  resulted  from  either  a formal  or  informal 
evaluation  of  the  technology's  operational  ef- 
fectiveness and  suitability.  We  improved  our 
devices  when  they  did  not  completely  satisfy 
their  intended  use.  Until  recently,  however,  this 
evaluation  process  tended  to  occur  too  late  in 
the  life  cycle  of  systems.  In  the  case  of  technol- 
ogy of  weapon  systems  especially,  we  waited  for 
training  or  combat  usage  to  show  us  the  flaws. 


Military  aviation  got  its  start  in  the  United 
States,  for  example,  not  on  30  July  1909,  at  Fort 


Myer,  Virginia,  when  the  Wright  Brothers  suc- 
cessfully demonstrated  their  Wright  Flyer,  but 
rather  sometime  later  when  Captain  Benjamin 
D.  Foulois  who,  in  the  course  of  teaching  him- 
self to  fly,  discovered  the  airplane’s  useful  capa- 
bilities and  its  operational  weaknesses.  Hard 
landings  often  dumped  the  135-pound  Foulois 
tp  the  ground.  To  rectify  the  problem,  Foulois 
mounted  a metal  tractor  seat  on  the  airplane. 
When  that  proved  less  than  satisfactory.  Cap- 
tain Foulois  looped  his  Sam  Browne  belt 
through  the  tractor  seat  and  strapped  it  tight 
around  his  lap.  Contoured  seats  and  safety  belts 
have  been  a part  of  the  aviation  scene  ever  since. 

More  than  30  years  later,  early  American 
participation  in  the  European  air  war  proved 
the  need  for  long  range  escort  fighters  to  ac- 
company strategic  bombers  to  the  target  area. 
Without  fighter  escort,  bomber  losses  were  pro- 
hibitive and  deep  penetration  of  the  enemy 
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homeland  was  risky  and  ineffective.  As  devel- 
oped, produced,  and  deployed,  the  P-38,  P-51, 
and  P-47  fighters  of  the  time  did  not  have  the 
endurance  to  go  as  far  as  was  needed.  The 
fighter  planes  were  modified  in  England  to 
carry  belly-mounted  auxiliary  gas  tanks  that 
were  fabricated  in  the  field.  The  deficiency  was 
recognized  from  operational  experience  and 
was  corrected  by  modification  in  the  field  after, 
rather  than  prior  to  production. 

By  the  decade  of  the  sixties,  the  cost  of 
correcting  deficiencies,  discovered  from  the  op- 
erational use  of  systems  after  they  were  de- 
ployed, became  excessive  and  resulted  in  unan- 
ticipated increases  in  life  cycle  cost.  Clearly, 
operational  test  and  evaluation  as  it  was  tradi- 
tionally performed  was  taking  place  too  late  in 
the  acquisition  cycle  to  control  either  the  cost 
or  the  military  utility  of  new  weapons. 

There  was  a growing  conviction  among 
students  of  the  acquisition  system  that  opera- 
tional test  and  evaluation  (OT&E)  needed  to  be 
performed  sooner  if  it  were  to  contribute  to  the 
effectiveness  of  procurement  practices.  The 
1970  report  of  the  Blue  Ribbon  Defense  Panel 
said,  for  instance  that,  “...to  be  effective,  OT&E 
must  be  a total  process,  using  all  appropriate 
methods  of  evaluation,  which  spans  the  entire 
life  cycle  of  a system  from  initial  requirement 
until  it  is  phased  out  of  the  operational  forces.”’ 

Respoi.dirg  to  that  recommendation, 
DOD  Directive  5000.3,  19  Jan  1973,  specified 
that, 

“...Test  and  evaluation  shall  be  commenced  as 
early  as  possible  and  conducted  throughout  the 
system  acquisition  process  as  necessary  to  assist  in 
progressively  reducing  acquisition  risks  and  in  asses- 
sing military  worth.” 

and  that, 

“...Acquisition  schedules  will  be  based,  inter 
alia,  upon  accomplishing  test  and  evaluation  mile- 
stones prior  to  the  time  that  key  decisions  which 
would  commit  significant  added  resources  are  to  be 
made."’ 


In  short,  the  basic  concept  of  operational 
test  and  evaluation  (OT&E)  has  not  changed 
significantly.  The  difference  now  is  in  what  we 
are  doing,  and  how  it  is  being  done,  and  most 
significantly,  when  it  is  being  done.  The  major 
objective  now  is  to  prove  the  military  utility  of 
new  weapons  before,  rather  than  after  they  are 
produced. 

BEGINNING  OF  AFTEC 

In  1973  the  Air  Force  Chief  of  Staff  di- 
rected the  establishment  of  the  Air  Force  Test 
and  Evaluation  Center  (AFTEC)  at  Kirtland 
Air  Force  Base,  New  Mexico.’  The  original 
manning  called  for  212  authorizations  (141 
officers,  25  enlisted,  46  civilians).  These  people 
were  assigned  to  the  AFTEC  headquarters  staff 
at  Kirtland  Air  Force  Base,  New  Mexico,  and 
as  AFTEC  test  directors  at  the  field  test  sites 
where  operational  test  and  evaluation  (OT&E) 
is  actually  conducted. 

In  its  4-year  existence,  AFTEC  has  be- 
come increasingly  involved  in  planning,  con- 
ducting, and  reporting  on  operational  test  and 
evaluation  for  new  weapon  systems.  Some  ex- 
amples are  the  F-15  and  the  A- 10  aircraft,  the 
Cobra  Dane  radar,  and  the  Air  Force  Satellite 
Communications  System.  In  this  time,  AFTEC 
has  taken  several  initiatives  and  is  working  with 
Headquarters  United  States  Air  Force  to  im- 
prove procedures  for  conducting  and  indepen- 
dently reporting  on  operational  test  and  evalua- 
tion. The  primary  areas  include:  providing  op- 
erational test  and  evaluation  in  as  realistic  an 
operational  environment  as  possible;  establish- 
ing levels  of  performance  for  measuring  a 
weapon  system’s  effectiveness;  and  assisting  in 
developing  and  formalizing  procedures  for  a 
new  deficiency  reporting  system  for  the  Air 
Force. 

The  Air  Force  Test  and  Evaluation  Center 
headquarters  manages  the  operational  test  and 
evaluation  on  new  major  weapon  systems  to 
include:  day-to-day  activities  of  ongoing  pro- 
grams, provisioning  for  resources  for  the  con- 
duct of  operational  test  and  evaluation,  and 
preparation  of  test  plans  and  reports.  The  head- 
quarters staff  also  supports  the  test  directors  at 
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the  field  test  sites.  At  the  field  test  sites,  the  field 
test  teams,  under  the  operational  control  of  an 
AFTEC  test  director,  execute  the  tests  and 
prepare  the  test  reports.  The  field  test  team 
consists  of  people  from  the  major  air  commands 
(MAJCOM)  who  will  operate  and  support  the 
system  when  it  is  deployed. 

For  example,  an  AFTEC  operating  loca- 
tion at  Edwards  Air  Force  Base,  California, 
consists  of  separate  test  teams  for  the  F-16  air- 
craft and  the  advanced  medium  short  take-off 
and  landing  transport.  Each  of  these  test  teams 
includes  operations-oriented  personnel  from 
the  using  command  and  personnel  from  the 
supporting  commands  (e.g.,  the  YC-14  test 
team  consists  of  Military  Airlift  Command,  Air 
Force  Logistics  Command,  and  Air  Training 
Command  personnel).*  In  each  case,  however, 
the  operational  test  and  evaluation  purpose  is 
the  same:  to  provide  an  independent  assessment 
of  how  well  a weapons  system  works  in  an 
operational  environment. 

THE  DT&E/OT&E  DISTINCTION 

Important  to  the  understanding  of  inde- 
pendent operational  testing  is  the  distinction 
between  development  test  and  evaluation 
(DT&E)  and  operational  test  and  evaluation 
(OT&Ey  The  fact  that  development  test  and 
evaluation  and  operational  test  and  evaluation 
necessarily  examine  the  same  features  of  a sys- 
tem is  evident  in  the  basic  definitions  of  DOD 
Directive  5000.3.  How  can  development  test 
and  evaluation  “demonstrate  that  design  is 
complete’*  without  also  estimating  “the  need 
for  modifications”  (an  operational  test  and 


•In  this  regard,  AFTEC  differs  from  the  Navy’s  Opera- 
tional Test  and  Evaluation  Force  (OPTEVFOR).  The 
OPTEVFOR  consists  of  a headquarters  in  Norfolk,  Vir- 
ginia, and  test  elements  that  conduct  OT&E  on  weapons 
systems  OPTEVFOR  actually  owns  and  operates.  The 
Army’s  Operational  Test  and  Evaluation  Agency,  OTEA, 
in  Falls  Church,  Virginia,  does  business  in  much  the  same 
way  as  does  AFTEC.  The  Operational  Test  and  Evalua- 
tion Agency  designs  and  conducts  operational  tests  for 
major  and  Category  I nonmajor  systems;  and  selectively 
reviews,  approves,  and  monitors  operational  tests  for 
nonmajor  system  testing  conducted  by  other  Army  testing 
agencies. 


evaluation  function)?  The  answer  is  it  cannot. 
Why  then  are  development  test  and  evaluation 
and  operational  test  and  evaluation  being  asked 
similar  questions?  The  reason  is  that  their  view- 
points are  completely  different. 

Development  test  and  evaluation  is  the  re- 
sponsibility of  Air  Force  Systems  Command 
whose  viewpoint  is  primarily  technical.  Pri- 
mary responsibilities  are  to  answer  the  ques- 
tions: (1)  does  the  system  meet  engineering  spe- 
cifications; and  (2)  is  the  system  sufficiently  de- 
veloped to  enter  production?  These  issues  are 
generally  satisfied  by  testing  in  a controIJed  en- 
vironment that  minimizes  the  chance  that  un- 
known (or  unmeasured)  variables  will  affect 
system  performance.  The  testing  is  conducted 
by  technical  personnel  skilled  at  fine-tuning  to 
maximize  performance. 

Operational  test  and  evaluation  is  priman- 
ly  the  responsibility  of  AFTEC  and  the  ucing 
MAJCOM  whose  viewpoint  is  operator-ori- 
ented. Operational  test  and  evaluation  issues 
are:  (1)  does  the  system  operate  effectively  in  its 
intended  operational  environment;  and  (2)  can 
“blue-suit”  personnel  maintain  the  system  as 
envisioned  in  the  maintenance  concept.  These 
responsibilities  are  best  satisfied  in  an  uncon- 
trolled environment  where  many  operational 
variables  may  affect  the  system  performance, 
e.g.,  operator  training  and  skill  level,  compati- 
bility and  interoperability  with  other  systems 
and  simulated  threat  environment.  Testing  is 
conducted  by  the  operators  and  maintenance 
personnel  who  will  use  the  system  in  the  field. 

An  example  from  the  E-3A  Airborne 
Warning  and  Control  System  (AW ACS)  pro- 
gram might  illustrate  the  difference  between  a 
developer’s  viewpoint  and  an  operator’s  con- 
cern. The  developer  might  satisfy  his  specifica- 
tions if  the  AW  ACS  surveillance  radar  func- 
tions properly  and  gives  a specific  detection 
range  at  a specific  altitude  (a  test  environment). 
The  operational  tester,  however,  would  also 
want  to  know  how  well  a radar  scope  operator 
and  the  radar  (under  the  same  conditions)  fol- 
low maneuvering  tracks  in  a hostile  electronics 
countermeasures  environment  after  the 
AWACS  has  been  aloft  and  operating  its  sur- 
veillance radar  for  a number  of  hours  (an  op- 
erational environment). 
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Such  an  example  illustrates  the  dilTerence 
between  development  test  and  evaluation  con- 
cerns. But,  the  example  does  not  mean  that  the 
developer  and  operational  tester  are  opposed  to 
each  other.  Rather,  development  test  and  evalu- 
ation and  operational  test  and  evaluation  com- 
plement each  other  and  form  one  path  toward 
the  same  goal — getting  an  effective  weapon  sys- 
tem for  the  Air  Force.  In  fact,  the  Defense 
Science  Board  recently  reaffirmed  the  close  re- 
lationship. 

The  Defense  Science  Board  encourages  a 
close  interaction,  “...particularly  feedback  from 
the  operational  test  and  evaluation  to  the  devel- 
oper. Interaction  among  development  test  and 
evaluation,  operational  test  and  evaluation,  and 
close  contact  with  the  user  pay  very  important 
dividends  in  terms  of  money,  time,  and  op- 
erational suitability.”*  Thus,  AFTEC  will  con- 
tinue to  support  testing  that  gives  both  the  user 
and  developer  required  information  and,  at  the 
same  time,  yields  valuable  operational  test  and 
evaluation  data  that  is  separately  analyzed  and 
reported  by  the  AFTEC  test  team. 

However,  as  indicated  earlier,  the  develop- 
ment test  and  evaluation  tester  and  the  opera- 
tional test  and  evaluation  tester  have  different 
viewpoints  of  a system.  This  difference  obli- 
gates AFTEC,  as  the  Air  Force  independent 
operational  test  and  evaluation  agency,  to  con- 
duct additional,  dedicated  operational  test  and 
evaluation.  Such  testing  gives  a better  estimate 
of  the  full  picture  of  a weapon  system’s  opera- 
tional capability.  Recent  findings  by  other  gov- 
ernment agencies  make  it  clear  that  such  dedi- 
cated testing  is  a keystone  in  the  weapon  system 
acquisition  cycle  before  the  Production  and  De- 
ployment Decision.  In  keeping  with  its  charter, 
AFTEC  independently  reports  its  operational 
test  and  evaluation  findings  directly  to  the  Air 
Force  Chief  of  Staff. 

IMPROVING  OT&E 

Thus  AFTEC  was  created  to  manage  the 
Air  Force’s  operational  test  and  evaluation  pro- 
gram. To  fulfill  this  charter,  AFTEC  embarked 
upon  a number  of  initiatives  to  more  closely  re- 
late operational  requirements  to  the  acquisition 
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of  weapon  systems  and  to  improve  the  opera- 
tional test  and  evaluation  process.  The  most  im- 
portant of  these  initiatives  deserve  some  elabo- 
ration; early  operational  test  and  evaluation  in- 
volvement in  the  acquisition  process;  opera- 
tional concept;  real-world  testing;  levels  of  sys- 
tem performance;  and  deficiency  reporting. 

EARLY  OT&E  INVOLVEMENT  IN  THE 
ACQUISITION  PROCESS 

As  part  of  its  effort  to  improve  operational 
testing,  AFTEC  now  gets  involved  as  early  as 
possible  in  the  acquisition  process,  as  far  back 
as  Milestone  0,  Program  Initiation.  These  early 
phases  of  the  acquisition  process  have  tradition- 
ally been  the  exclusive  realm  of  development 
engineers.  Why,  then,  is  AFTEC  concerned 
about  early  involvement?  Well,  for  several  im- 
portant reasons.  First,  so  that  developers  and 
contractors  can  learn  very  early  of  specific 
operational  considerations  against  which  the 
system  will  be  tested.  Second,  so  that  contrac- 
tors and  developers  will  have  a much  clearer 
idea  of  the  resources  needed  to  conduct  opera- 
tional test  and  evaluation.  Finally,  AFTEC  can 
plan  early  for  realistic  tests  that  will  provide 
answers  to  important  operational  questions 
voiced  during  the  decisionmaking  process,  in- 
cluding those  asked  by  members  of  the  Defense 
Systems  Acquisition  Review  Council 
(DSARC). 

The  MX  strategic  missile  is  an  excellent 
example  of  this  early  involvement.  The  MX  is 
still  in  the  Demonstration  and  Validation 
Phase.  Nonetheless,  AFTEC  has  operational 
test  and  evaluation  test  personnel  working  with 
the  System  Program  Office  at  Norton  Air 
Force  Base,  California.  Even  though  actual 
MX  launches  will  probably  not  take  place  until 
early  1980,  AFTEC  involvement  now  will 
mean  that  operational  considerations  are  inte- 
grated into  the  development  and  subsequent 
testing  of  the  system.  The  AFTEC  contingent 
will  form  the  nucleus  of  the  future  test  team 
that  will  conduct  operational  test  and  evalua- 
tion on  full-scale  MX  missiles. 

To  identify  future  programs  for  early 
AFTEC  involvement,  a Planning  Division  has 
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been  established  at  Headquarters,  AFTEC. 
This  division  works  with  the  appropriate  Sys- 
tem Program  Office  and  the  Air  Staff  to  insure 
that  operational  test  and  evaluation  require- 
ments are  included  in  early  program  documen- 
tation and  to  make  sure  that  an  operational  or 
real-world  flavor  is  included  in  the  test  program 
from  the  very  beginning. 

OPERATIONAL  CONCEPT 

Closely  related  to  the  early  start  of  opera- 
tional test  and  evaluation  is  the  requirement  for 
an  operational  concept  for  a new  weapon  sys- 
tem. The  operational  concept  describes  how  the 
weapon  system  will  be  employed  by  the  opera- 
tor. We  have  been  working  closely  with  Head- 
quarters, United  States  Air  Force  in  writing  an 
Air  Force-wide  operational  concept  regulation. 
Basically,  this  regulation  will  require  the  using 
major  air  command  to  clearly  define  early  in 
the  acquisition  cycle  the  employment,  deploy- 
ment, and  support  concepts  for  a system. 

The  operational  concept  will  be  a living 
document.  As  tactics  or  technology  change,  the 
concept  will  be  updated.  The  document  will 
provide  a basis  for  system  test  and  evaluation. 
In  addition,  it  will  provide  for  such  items  as  the 
posturing  of  combat  forces  and  standards  for 
basing  forces  and  equipment.  In  short,  this 
user-written  operational  concept  will  be  a basis 
for  establishing  specific  operational  objectives 
and  evaluation  criteria.  Hence,  AFTEC  will  be 
able  to  test  against  these  criteria  and  estimate 
the  effectiveness  of  the  system  early  in  the 
acquisition  cycle. 

REAL-WORLD  TESTING 

Early  in  this  article,  mention  was  made  of 
devdopment  test  and  evaluation  and  opera- 
tional test  and  evaluation, tind  the  differences 
between  them.  The  skilled  engineers  and  the 
controlled  conditions  of  development  testing 
must  eventually  give  way  to  blue-suit  mainte- 
nance and  testing  under  operational  conditions 
whether  the  system  evaluated  be  a radar,  a 
communications  network,  or  a fighter  aircraft. 
In  other  words,  development  test  and  evalua- 
tion, conducted  in  a controlled  environment. 


must  give  way  to  operational  test  and  evalua- 
tion conducted  in  an  operational  environment. 
The  ultimate  test  of  a weapon  system’s  opera- 
tional effectiveness  and  suitability  must  be  in 
the  operational  or  real-world  environment. 

Two  examples  of  real-world  testing  and  a 
discussion  of  some  steps  AFTEC  has  taken  to 
increase  its  interface  with  the  users  of  new 
weapon  systems  will  illustrate  the  point.  First, 
in  November  1976,  AFTEC,  working  with  Tac- 
tical Air  Command,  Strategic  Air  Command, 
and  Aerospace  Defense  Command,  tested  the 
E-3A  Airborne  Warning  and  Control  System  in 
the  largest  opposing  air  operation  (in  terms  of 
time  and  space  density)  ever  conducted  during 
peacetime.  Over  400  aircraft  from  airbases  in 
the  western  United  States  flew  over  the 
Edwards/Nellis  range  (located  in  Southern 
California  and  Nevada)  in  a simulated  air  battle 
between  Warsaw  Pact  nations  and  NATO.  A 
production  E-3A  was  tested  as  it  performed  its 
full  command,  control,  and  surveillance  func- 
tions during  the  exercise.  Only  an  actual  com- 
bat situation  could  have  been  more  realistic. 

A second  example,  the  May  1977  F-IS 
European  Tactical  Test,  shows  that  operational 
testing  is  not  necessarily  a lengthy  process.  The 
Air  Force  Chief  of  Staff  wanted  an  operational 
evaluation  of  the  F-15  in  Europe.  In  less  than  2 
months,  AFTEC,  working  closely  with  Tactical 
Air  Command  and  United  States  Air  Forces  in 
Europe,  planned,  flew,  and  reported  on  the  F- 
15  in  Europe.  Eight  specific  missions  were 
flown  against  a simulated  Warsaw  Pact  threat. 
This  real-world  operational  test  and  evaluation 
yielded  valuable  information,  within  peacetime 
limitations,  about  the  F-15,  its  fire  control  sys- 
tem and  weapons,  and  the  aircraft  effectiveness 
in  the  European  environment. 

To  increase  the  opportunities  for  such  real- 
world  testing,  AFTEC  has  established  two  de- 
tachments, Detachment  #1  at  Ramstein  Air 
Base,  Germany,  and  Detachment  #2  at  Eglin 
Air  Force  Base,  Florida.  The  Ramstein  detach- 
ment will  accomplish  two  goals:  (1)  future 
operational  tests,  like  the  F-15  test,  will  be 
planned  and  coordinated  with  US  forces  in 
Europe;  and,  (2)  operational  test  and  evaluation 
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conducted  in  the  US  will  always  have  a “think  out  realistic  evaluation  criteria  for  specific  test 

Europe”  flavor  because  the  AFTEC  European  objectives.  But,  before  specific  criteria  could  be 

detachment  will  review  AFTEC  Operational  worked  for  each  system,  AFTEC  and  the  major 

Test  and  Evaluation  plans  to  assure  realism.  air  command  had  to  agree  on  a standardized 

The  Eglin  detachment  will  work  closely  with  terminology.  The  three  key  terms,  "thresh- 

the  Armament  Development  Test  Center  and  olds,"  "standards,"  and  “goals,"  are  currently 

the  Tactical  Air  Warfare  Center.  In  particular,  being  defined  and  used  in  on-going  programs. 

AFTEC  will  establish  liaison  with  the  Arma- 

ment  Development  Test  Center  and  the  Tacti-  Thresholds  are  quantitative  or  qualitative 

cal  Air  Warfare  Center  in  the  areas  of  opera-  minimum-essential  levels  of  performance/capa- 

tional  test  and  evaluation  reliability,  maintaina-  bility  that  permit  mission  accomplishment, 

bihty,  and  supportability  for  weapon  systems.  jbese  levels  are  based  on: 


The  concern  for  meaningful  operational 
test  and  evaluation  in  simulated  combat  situa- 
tions will  become  increasingly  important  in  the 
future.  In  the  1974-77  time  frame,  AFTEC  was 
primarily  testing  new  types  of  hardware  (e.g., 
the  F-15,  the  A- 10,  and  the  AIM-7F  missile) 
that  are  coming  into  the  Air  Force  inventory. 
The  next  several  years  of  operational  test  and 
evaluation  will  continue  such  testing  but  there 
will  be  more  concentration  on  operational  test 
scenarios  (such  as  the  Airborne  Warning  and 
Control  System  tactical  test)  that  permit  testing 
of  refinements  and  enhancements  to  these 
systems. 

LEVELS  OF  SYSTEM  PERFORMANCE 

In  the  past,  AFTEC  was  all  too  frequently 
asked,  “What  criteria  did  you  test  the  system 
against?”  In  some  cases,  AFTEC  had  to  admit 
that  evaluations  were  too  subjective  in  nature. 
What  was  missing  were  clearly-defined  opera- 
tional test  and  evaluation  criteria  against  which 
to  test  new  weapon  systems.  The  Air  Force  Test 
and  Evaluation  Center  looked  at  program  doc- 
umentation for  each  new  system  (e.g..  Required 
Operational  Capability,  Decision  Coordinating 
Paper,  Program  Management  Directive,  and 
operationally  significant  values  contained  in 
contractual  specifications,  etc.).  However, 
these  documents  are  issue-oriented  or 
engineering-oriented  and  do  not  always  yield 
meaningful  operational  test  and  evaluation 
criteria. 

The  Air  Force  Test  and  Evaluation  Center 
then  started  working  with  the  using  major  air 
command  of  each  new  weapon  system  to  spell 


• operational  and  maintenance  concepts; 

• the  threat  estimate; 

• operationally  significant  performance 
contained  in  documents  such  as  the 
Decision  Coordinating  Paper  (DCP), 
Program  Management  Document 
(PMD),  Test  and  Evaluation  Master 
Plan  (TEMP),  etc.;  and 

• the  capabilities  of  existing  systems 
(when  a valid  comparison  can  be 
made). 

Standards  are  quantitative  or  qualitative 
levels  of  performance/capability  that  will  sat- 
isfy the  operational  requirements  established  at 
Milestone  II  for  a fully  operational  system. 
These  levels  are  based  on: 

• the  threat  estimate; 

• operational  and  maintenance  concepts 
that  are  approved  at  Milestone  II;  and 

• other  program  documentation  when 
appropriate. 

Goals  are  quantitative  or  qualitative  levels 
of  performance/capability  that  will  enhance  the 
system  or  are  new  requirements  identified  after 
Milestone  II.  These  levels  are  based  on: 

• operationally  significant  performance 
contained  in  documents  such  as  the 
DCP,  PMD,  TEMP,  etc.; 
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• identification  of  an  increase  in  the 
threat  estimate;  and 


• changes  in  the  operational  and  mainte- 
nance concepts  formulated  at  Mile- 
stone II. 


Further,  goals  can  be  identified  as  additional 
system  attributes  or  logistics  support  ap- 
proaches not  contracted  for  that,  if  achievable 
at  a reasonable  cost,  will  result  in  significant 
improvement  to  the  system. 


Performance  below  the  thresholds  is  defi- 
cient. This  requires  management  action  to  raise 
the  level  of  performance  to  permit  mission 
accomplishment. 

An  example  of  how  thresholds,  standards, 
and  goals  were  used  as  test  criteria  on  the  F-4G 
Wild  Weasel  will  illustrate  the  value  of  this  new 
approach  to  operational  test  and  evaluation. 
One  important  Wild  Weasel  operational  perfor- 
mance parameter  was  emitter  detection.  The  F- 
4G  contract  specification  called  for  detection  of 
any  given  number  of  threat  radars  within  “x” 
number  of  seconds — but  failed  to  spell  out  what 
percentage  of  time  the  F-4G  should  achieve  this 
mark.  The  Air  Force  Test  and  Evaluation  Com- 
mand, working  closely  with  the  Tactical  Air 
Command,  looked  at  the  threat,  the  expected 
combat  scenario,  and  how  the  F-4G  system 
operated.  Based  on  this  analysis  and  the  current 
operational  requirement  of  the  F-4G,  AFTEC 
and  Tactical  Air  Command  established  thresh- 
olds, standards,  and  goals  for  the  Weasel’s 
radar  detection  rate.  Thus,  based  on  the  opera- 
tional test,  the  F-4G  crew  knew  how  to  grade 
the  system  performance.  For  example,  if  the 
system  had  exhibited  only  the  threshold  per- 
formance (detection  of  a certain  percentage  of 
all  emitter  signals  in  “x”  number  of  seconds), 
an  aircrew  would  know  there  were  emitters 
they  were  not  detecting  and  fast  movement  was 
needed  to  avoid,  say,  a simulated  surface-to-air 
missile  threat.  If  the  system  failed  to  reach  the 
threshold,  it  would  be  rated  deficient,  and  the 
deficient  hardware  or  tactics  would  have  to  be 
corrected. 

This  evaluation  criteria  approach  to  opera- 
tional test  and  evaluation  is  not  exactly  new. 


Thresholds,  standards,  and  goals  were  con- 
tained in  the  contract  with  the  Wright  Brothers 
for  the  first  military  airplane  procured  by  the 
United  States  government  in  1908.*  The  gov- 
ernment wanted  a heavier  than  air  machine 
that  could  reach  40  miles  per  hour  in  still  air 
conditions.  The  “t/iresAoJd ’’speed  was  36  miles 
per  hour,  the  “standard”'*/as  40  miles  per  hour, 
and  the  "goal”  (the  attainment  of  which  would 
result  in  a bonus  payment)  was  44  miles  per 
hour. 

DEFICIENCY  REPORTING 

When  a piece  of  hardware  fails  to  perform 
satisfactorily  during  the  test  program,  a defi- 
ciency report  is  written  by  the  test  team.  The 
Air  Force  Test  and  Evaluation  Command  has 
been  working  with  the  major  air  commands  and 
the  Air  Staff  to  put  deficiency  reporting  in  tune 
with  the  peculiarities  of  Air  Force  test  and 
evaluation.  This  has  resulted  in  a section  in  the 
new  Technical  Order  00-35D-54,  The  USAF 
Material  Deficiency  Reporting  and  Investigat- 
ing System,  which  lays  out  separate  deficiency 
reporting  procedures  to  be  followed  during  test 
and  evaluation.  As  part  of  the  implementation 
of  this  system,  deficiencies  report^  to  the  Sys- 
tem Program  Office  are  now  “prioritized,”  first 
by  the  urgency  of  the  need  for  the  correction 
(e.g.,  mission  essential  or  mission  degrading), 
and,  then,  by  periodic  rank  ordering.  Efforts 
are  aimed  at  focusing  the  attention  of  Program 
Managers  on  the  key  deficiencies  that  must  be 
corrected  to  make  the  weapon  system  effective 
in  the  operational  environment.  Deficiency 
“prioritization”  is  definitely  not  a new  concept,  i 

but  is  receiving  added  emphasis  today. 

The  Air  Force  Test  and  Evaluation  Com-  j 

mand  is  also  placing  a great  deal  of  emphasis  on 
tracking  the  status  of  deficiency  corrections.  ' 

Deficiency  reports  are  tracked  by  the  AFTEC 
test  team  from  the  time  of  discovery  of  the 
deficiency  through  determination  of  the  proper 


•Terence  R.  St.  Louis,  AFTEC  historian,  has  a copy  of 
the  contract,  dated  10  Feb  1908,  between  the  Signal 
Corps,  United  States  Army,  and  the  Wright  Brothers.  The 
contract  specifies  minimum,  acceptable,  and  superior  per- 
formance goals — and  appropriate  payments  thereto. 
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corrective  action  by  the  development  engineers 
to  physical  verification  of  the  “fix”  by  the  test 
team  or  the  using  command.  The  purpose  is  to 
provide  using  command  and  AFTEC  managers 
an  up-to-date,  accurate  status  accounting  of 
deficiency  corrective  actions.  This  will  help 
identify  important  corrective  actions  that  may 
require  using  command  or  AFTEC  manage- 
ment attention  for  timely  resolution. 

The  recently  concluded  A- 10  follow-on 
test  and  evaluation  was  used  as  a test  case  for 
the  new  Deficiency  Report  section  of  Technical 
Order  00-35D-54.  The  results  were  that  defi- 
ciency reports  were  standardized  and  were 
“tracked”  more  easily  from  originator  to  the 
System  Program  Office  responsible  for  correct- 
ing the  deficiencies.  Also,  because  the  defi- 
ciency reports  were  in  order  of  priority,  man- 
agers focused  attention  on  the  key  deficiencies 
right  away. 

OTHER  AREAS 

The  Air  Force  Test  and  Evaluation  Com- 
mand is  seeking  to  improve  operational  test  and 
eva'uation  in  several  other  areas.  As  weapon 
systems  have  become  increasingly  complex,  so, 
too,  have  the  related  subsystems.  In  this  regard, 
we  are  now  directly  involved  in  the  operational 
test  and  evaluation  of  computer  sohware  and 
simulator  programs-dne  former  as  they  affect 
computer  controlled  programs,  weapons,  and 
command  control  communications,  the  latter 
because  of  the  increased  emphasis  being  given 


simulators  to  augment  training.  The  Air  Force 
Test  and  Evaluation  Command  is  applying  a 
computer  simulation  model,  the  Logistics 
Composite  Model,  to  determine  maintenance 
manpower  requirements  for  new  aircraft  sys- 
tems. The  model  “predicts”  resources  (people, 
parts,  facilities)  needed  to  support  a weapon 
system  over  a given  period  of  time  and  the 
impact  of  resource  shortages  on  the  operational 
status  of  a flying  unit.  Finally,  AFTEC  is  devel- 
oping an  OT&E  Management  Document  that 
will  provide  guidance  on  how  to  plan,  conduct, 
and  report  on  operational  test  and  evaluation. 
It  will  serve  to  standardize  operational  test  and 
evaluation  throughout  the  Air  Force. 


SUMMARY 

In  summary,  then,  AFTEC  is  a young, 
independent  Air  Force  agency  that  is  in  dy- 
namic evolution  in  its  efforts  to  improve  Air 
Force  operational  test  and  evaluation.  The  Air 
Force  Test  and  Evaluation  Command  is  pursu- 
ing its  charter  on  two  fronts.  First,  it  is  actively 
seeking  early  consideration  of  operational  test 
and  evaluation  requirements  in  the  acquisition 
cycle  even  before  prototype  hardware  is  built. 
Second,  as  the  lead  Air  Force  operational  test 
and  evaluation  agency,  AFTEC  is  continually 
pursuing  new  initiatives  to  improve  Air  Force 
operational  test  and  evaluation.  These  all  add 
up  to  what  AFTEC  feels  is  the  bottom  line — to 
ensure  the  US  Air  Force  is  getting  the  most 
effective  weapon  systems  possible. 
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OPERATIONAL  TEST  AND  EVALUATION 
OF  SHIPS:  POLICY  AND  PRACTICE 
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and 
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The  primary  reason  for  conducting  operational  test  and  evaluation  (OT&E)  in  major  ship 
acquisition  programs  is  to  reduce  operational  risks  rapidly  and  minimize  the  need  for 
modification.  The  second  reason  for  conducting  operational  test  and  evaluation  is  to  gain 
insights  into  tactical  employment  of  the  new  warship.  This  is  particularly  important  when  new 
warfare  capabilities  are  introduced,  as  in  NATO  hydrofoil  and  AEGIS  ships.  In  this  paper  the 
authors  discuss  ship  operational  test  and  evaluation,  the  policy  behind  it,  and  the  ways  in  which 
it  is  carried  out  in  practice.* 


MODERN  WARSHIP 
COMPLEXITY 


A modern  warship — an  aircraft  carrier,  a 
cruiser,  a submarine — is  the  most  complex,  self- 
contained,  highly  automated  and  rugged  as- 
sembly of  equipment  constructed  by  man.  She 
must  be  capable  of  steaming  and  fighting  inde- 
pendently or  in  coordination  with  other  forces, 
including  Allied  Forces.  Whether  on  or  below 
the  surface  of  the  ocean,  she  must  be  at  home  in 
the  environment,  even  when  that  environment 
is  turbulent,  as  it  frequently  is.  She  must  carry 
with  her  the  men  and  stores  to  operate,  main- 
tain and  support  her  weapons,  sensors,  propul- 
sion machinery  and  auxiliary  systems,  far  from 
shorebased  support.  When  fuel,  ammunition, 
or  food  are  needed,  she  must  be  capable  of 
receiving  them  at  sea,  while  maintaining  her 
battle  readiness.  She  must  be  able  to  communi- 
cate classified  and  unclassified  information 
over  vast  distances  and  she  must  be  able  to 
detect,  identify,  locate  and  attack  a variety  of 


•Much  of  the  material  about  the  practice  of  ship  opera- 
tional test  and  evaluation  has  been  taken  from  oflicial 
COMOPTEVFOR  evaluation  reports  to  the  CNO. 


air,  surface  and  subsurface  threats  before  she 
herself  is  attacked.  For  these  reasons,  a modern 
warship  is  an  integration  of  practically  all 
military /technical  disciplines,  and  incorporates 
a complexity  unthinkable  20  or  30  years  ago. 
While  this  complexity  is  the  source  of  her 
capabilities,  it  also  causes  many  of  her  difficul- 
ties. Complexity  increases  the  time  required  for 
ship  design  and  construction;  it  magnifies  relia- 
bility and  maintainability  requirements;  and  it 
escalates  the  need  for,  and  cost  of,  test  and 
evaluation. 

POLICY  FOR  OT&E  IN  SHIP 
ACQUISITION  PROGRAMS 

Warship  complexity  is  recognized  in  De- 
partment of  Defense  (DOD)  and  Navy  acquisi- 
tion directives.  The  test  and  evaluation  policy 
contained  in  these  directives  is  a codification  of 
lessons  learned  in  the  prior  shipbuilding  pro- 
grams, incorporating  the  best  elements  of  suc- 
cessful ship  acquisitions  and  applying  them  to 
future  programs. 


j 
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Department  of  Defense  Directive  5000.1, 
“Major  System  Acquisitions,”  is  the  corner- 
stone of  the  policy,  stipulating  that  “...Pro- 
grams shall  be  structure  and  resources  allo- 
cated to  ensure  that  the  successful  demonstra- 
tion of  program  objectives  is  the  pacing  activ- 
ity...,” and  requiring  that  test  and  evaluation 
begin  as  early  as  possible. 

Within  the  Navy,  DOD  Directive  5000.1 
was  implemented  by  SECNAV  Instruction 
5000. 1,  “System  Acquisition  in  the  Department 
of  the  Navy.”  In  this  Instruction,  the  Comman- 
der, Operational  Test  and  Evaluation  Force 
(COMOPTEVFOR),  under  the  command  of 
the  Chief  of  Naval  Operations  (CNO),  is  as- 
signed responsibilities  as  the  Navy’s  indepen- 
dent test  agency  for  the  required  operational 
test  and  evaluation  (OT&E). 


A Memorandum,  subject:  “Ship  Program 
Procedures,”  issued  by  the  Deputy  Secretary  of 
Defense  on  30  September  1975  sets  forth  the 
unique  sequence  of  program  events  and  mile- 
stones for  structuring  major  ship  acquisition 
programs  to  comply  with  DOD  Directive 
5000.1.  The  memorandum  establishes  the  stan- 
dard procedure  for  communication  of  plans 
and  decisions  about  the  Navy’s  overall  ship 
design,  development,  and  acquisition  pro- 
grams.* The  Navy  initiates  the  communication 
at  the  beginning  of  each  fiscal  year  with  an 
estimate  of  all  POM  (Program  Objectives 
Memorandum)  ship  development  and  acqui- 
sition programs,  and  the  key  dates  (including 
operational  test  and  evaluation  events),  prelimi- 
nary cost  estimates,  recommended  milestones, 
and,  if  necessary,  rationale  for  variance  from 
the  prescribed  program  events  shown  below: 


Ship  Acquisition  Program  Events 

Item 

Normally  authorized 
at  Milestone 

Initiate  preliminary  design 

I 

Initiate  contract  design;  long  lead 
time  procurement  for  lead  ship 

II 

Lead-ship  detailed  design  and  construction; 

Follow-ship  long  lead  time  procurement 

III 

Follow-ship  construction 

IIIA 

I 1 

If  a prototype  ship  is  to  be  developed,  typical  program  events  are: 

Prototype  ship  detailed  design 
and  construction 

IIA 

Follow-ship  construction  (if  any) 

III 

•"System  Acquisition  in  the  Department  of  the  Navy," 
(draft)  SECNAVINST  5000.IA,  directs  that  ship  pro- 
grams designated  as  major  be  structured  in  accordance 
with  the  30  Sep  75  DOD/Navy  agreement,  as  well  as 
DOD  Directive  5000. 1 . 


Vol  I,  No.  5 


29 


Department  of  Defense  Directive  5000.3, 
"Test  and  Evaluation,”  specifies  two  funda- 
mental types  of  operational  test  and  evaluation: 
initial  operational  test  and  evaluation  that  pre- 
cedes Milestone  111,  and  follow-on  operational 
test  and  evaluation  that  succeeds  Milestone  111. 
Department  of  Defense  Directive  5000.3  speci- 
fies a test  and  evaluation  policy  for  ships  that  is 
different  from  other  system  acquisitions.  Rec- 
ognition of  ship  complexity,  the  long  con- 
struction times,  and  the  need  for  a different 
management  approach  dictated  the  policy. 
Provision  is  made  for  progressive  steps — 
development,  test,  engineering,  integration,  and 
more  test — of  complex  combat  systems  at  a 
land-based  test  site,  before  the  major 
production  decision  for  the  ship  class.  For  ad- 
vanced nonnuclear  ship  propulsion  systems, 
adequate  test  and  evaluation  of  prototypes  is 
required  prior  to  the  first  major  production 
decision.  (Test  and  evaluation  of  Navy  nuclear 
propulsion  plants  is  accomplished  in  accor- 
dance with  methods  in  use  by  the  Department 
of  Energy).  Follow-on  test  and  evaluation  on 
the  lead  ship  of  a new  class  is  prescribed  for 
specified  systems  and  equipments,  and  a new 
concept  is  introduced:  "...if  required,  full  ship 
operational  evaluation  to  the  degree  feasible.”' 
Within  the  Navy  DOD  Directive  5000.3  is 
implemented  through  OPNAV  Instruction 
3960. 10,  "Test  and  Evaluation.” 

PLANNING  FOR  OT&E  IN  SHIP 
ACQUISITION  PROGRAMS 

In  the  last  5 years,  the  Navy  has  invested 
heavily  in  technical  and  operational  testing 
with  emphasis  on  structuring  milestone-ori- 
ented ship  acquisition  programs.  Individual 
shipboard  equipments  are  being  subjected  to 
technical  evaluation  by  the  developing  Agency 
and  extensive  operational  evaluation  by 
COMOPTEVFOR  before  receiving  approval 
for  service  use.  The  approval  for  service  use 
process  has  been  strengthened  and  directly  tied 
to  operational  test  and  evaluation.  Achieve- 
ment of  approval  for  service  use  is  now  an  inte- 
gral step  in  selecting  new  equipments  for  incor- 
poration into  warship  designs.* 

Land-based  test  sites,  useful  for  both  de- 
velopment test  and  evaluation  and  operational 


test  and  evaluation  on  new  ship  propulsion 
systems  and  combat  systems,  are  an  important 
engineering  tool  in  many  ship  acquisition  pro- 
grams. The  lead  ship  of  a new  class  now  is  being 
subjected  to  far  more  comprehensive  opera- 
tional test  and  evaluation  after  delivery  to  the 
government. 

The  top  level  management  tool  used  for 
planning  the  various  phases  and  types  of  test 
and  evaluation  is  the  Test  and  Evaluation 
Master  Plan  (TEMP).  The  TEMP  is  the  single 
Navy  document  that  integrates  development 
test  and  evaluation,  operational  test  and  evalua- 
tion, and  production  acceptance  test  and  evalu- 
ation for  a ship  acquisition  program.  In  accor- 
dance with  the  Navy  governing  directive  for 
TEMP,  OPNAVINST  3960.10,  planning  starts 
early.  The  TEMP,  at  least  in  outline,  should  be 
drafted  before  Milestone  I,  and  have  received 
approval  from  the  Chief  of  Naval  Operations  by 
Milestone  II. 

The  TEMP  planning  and  coordination 
process  is  long  and  iterative — it  must  be,  be- 
cause of  the  uniqueness  and  complexity  of  the 
acquisition  process  for  a major  new  class  of 
warship.  A ship  TEMP  concentrates  on  ship- 
related  issues— design,  engineering,  and  inte- 
gration of  the  nonnuclear  propulsion  system 
and  the  combat  system,  including  the  ship’s 
operational  performance  in  specific  warfare 
missions.  The  ship  TEMP  identifies  only  the 
separate  equipment  and  subsystem  develop- 
ment programs  and  the  respective  TEMP  for 
these  items.  The  ship  TEMP  then  can  be  fo- 
cused on  relevant  ship-level  issues.  Reference  is 


•OPNAVINST  4720.9D,  page  4,  sets  forth  the  concept  of 
approval  for  service  use  and  criteria  for  decisionmaking. 
Approval  for  service  use  is  “...that  determination  made  by 
the  Chief  of  Naval  Operations,  or  other  delegated  author- 
ity, that  new  systems  or  equipments  or  significant  alter- 
ations to  exi.sting  systems  or  equipments  have  undergone 
appropriate  test  and  evaluation,  to  the  extent  that  there 
has  been:  (1)  demonstrated  reliable  performance,  in  accor- 
dance with  design  specifications,  in  the  intended  or  exist- 
ing operational  environment;  (2)  demonstrated  ability  to 
be  operated  and  maintained  by  personnel  with  the  level  of 
skill  anticipated  to  be  available  under  Navy  service  condi- 
tions; and  (3)  sufficient  evidence  that  the  equipment  can 
be  supported  iogi.stically  in  a deployed  status." 
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made  to  the  component  subsystem-level  TEMP 
for  subsystem-level  details. 

The  TEMP  is  prepared  jointly  by  the  Ship 
Acquisition  Project  Manager  and  the  Comman- 
der, Operational  Test  and  Evaluation  Force 
(COMOPTEVFOR).  The  TEMP  is  approved 
by  the  Chief  of  Naval  Operations.  The  TEMP 
specifies  what  testing  will  be  done  when,  and 
what  resources  are  necessary  to  carry  out  the 
tests.  Approval  of  the  TEMP  constitutes  Chief 
of  Naval  Operation  direction  to  conduct  the 
test  and  evaluation  program  defined  therein,  in- 
cluding the  commitment  of  fleet  services  to  sup- 
port TEMP.  In  addition  to  a separate  status 
listing  of  all  related  subsystem  TEMP,  the  ship 
TEMP  includes  a separate  “Fleet  Introduction 
Schedule.”  This  schedule  highlights  the  many 
tests,  trials,  and  training  actions  that  must  be 
accomplished  on  the  lead  ship  before  it  can  be 
released  for  unrestricted  service.  The  schedule 
provides  a focal  point  for  the  ship  designers, 
builders,  testers,  trainers,  and  users  to  begin  co- 
ordinating the  most  critical  and  hectic  phase  in 
a new  warship’s  life,  her  fleet  debut.  The  “Fleet 
Introduction  Schedule”  and  the  considei  ations 
addressed  in  its  preparation  are  vital  to  conduc- 
ting orderly  operational  test  and  evaluation  on 
the  lead  ship. 

The  Commander,  Operational  Test  and 
Evaluation  Force  organizes  for  ship  operational 
test  and  evaluation  in  a manner  that  parallels 
the  ship  acquisition  structure.  The  separate 
nature  of  ship  sensor,  weapon,  and  propulsion 
machinery  development  programs,  and  the 
management  need  for  a perspective  of  the  total 
warship  acquisition  program  are  recognized.  • 
As  a matter  of  routine,  operational  test  direc- 
tors from  the  COMOPTEVFOR  staff  divisions 
(Underseas,  Air,  Command  and  Control,  and 
Surface  Warfare)  are  assigned  to  plan,  conduct, 
and  report  on  operational  test  and  evaluation 
for  individual  equipment.  An  additional  opera- 
tional test  director  from  the  Ship  Evaluation 
Division  of  the  staff  is  assigned  to  plan,  con- 
duct, and  report  on  overall  ship  operational  test 
and  evaluation.  Concern  is  with  the  ship  acquis- 
ition program  and  its  objectives.  The  opera- 
tional test  director  oversees  the  work  of  the 
operational  test  directors  who  are  evaluating 


individual  systems  that  will  eventually  be  inte- 
grated into  the  new  ship  design.  The  ship  opera- 
tional test  director  coordinates  with  the  “ship 
team”  to  ensure  that  all  relevant  ship  and 
equipment  operational  issues  are  addressed  at 
the  proper  level  in  a TEMP.  This  procedure  is 
not  easy,  but  it  is  essential,  because  the  saving  in 
expensive  resources  used  in  operational  test  and 
evaluation  can  be  significant.  For  example,  a 
single  firing  of  an  advanced  surface-to-air  mis- 
sile at  a drone  aircraft  can  generate  data  to  meet 
the  needs  of  the  operational  test  director  evalu- 
ating missile,  fire  control,  combat,  electronic 
warfare,  and  data  link  systems,  while  also  serv- 
ing the  ship  evaluation  operational  test 
director. 

The  operational  test  director  is  responsible 
for  preparing  the  input  of  the  Commander, 
Operational  Test  and  Evaluation  Force,  to  the 
TEMP.  Specifically,  this  test  director  is  ex- 
pected to: 

• Define  areas  of  operational  risk  for 
critical  test  and  evaluation  issues. 

• Delineate  the  performance  parameters 
by  which  operational  effectiveness  and 
operational  suitability*  will  be  assessed 
through  OT&E  before  each  milestone. 

• Prepare  the  “OT&E  Outline,”  that  in- 
cludes for  each  milestone/test  phase, 
the  operational  test  and  evaluation  ob- 
jectives, events/scope  of  testing/basic 
scenarios,  and  scope  of  the  operational 
test  and  evaluation  effort. 

• Determine  the  resources  needed  for 
operational  test  and  evaluation  and  re- 
source criticality  to  the  testing. 

TYPES  OF  SHIP  OT&E 

Ship  operational  test  and  evaluation  varies 
according  to  the  type  of  ship  development  and 
acquisition  program  and  according  to  when 


•Operational  suitability  includes  reliability,  maintainabil- 
ity, availability,  logistics  supportability,  compatibility,  in- 
teroperability. and  training  requirements. 
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(during  the  development  and  acquisition  pro- 
gram) it  is  conducted. 


There  are  two  primary  types  of  ship  ac- 
quisition programs: 

1 . Lead-ship,  follow-ship  program 

2.  Prototype  ship  program 

In  a lead-ship,  follow-ship  program,  the 
acquisition  strategy  is  structured  to  permit  ade- 
quate test  and  evaluation  of  new  subsystems 
ashore  and  at  sea  in  surrogate  ships,  nrior  to  re- 
lease of  funds  for  the  first  follow-ship  construc- 
tion (Milestone  IIIA).  This  strategy  represents 
a compromise  between  early  operational  test 
and  evaluation  on  a prototype,  and  late  opera- 
tional test  and  evaluation  on  the  first 
production  ship,  when  follow  ships  will  be  de- 
livered in  rapid  succession.  The  lead-ship, 
follow-ship  program  structure  provides  funding 
for  the  lead  ship  in  year  “1”,  no  funding  in  year 
“2”,  and  follow-ship  funding  in  years  “3”  and 
beyond.  This  schedule  permits  the  lead  ship  to 
undergo  continuing  phases  of  operational  test 
and  evaluation  to  and  through  fleet  introduc- 
tion, by  which  time  all  ships  of  the  class  may  be 
under  contract.  While  some  concurrency  of  de- 
velopment and  production  exists,  there  is  time 
for  lessons  learned  and  necessary  modifications 
identified  in  operational  test  and  evaluation  and 
incorporated  in  the  lead  ship,  to  be  fed  back 
into  follow  ships  while  these  ships  are  under 
construction.  Examples  of  lead-ship  follow-ship 
acquisitions  are  the  FFG  7 and  DDG  47 
classes,  each  having  a funding  profile  of  1, 0,  X, 
X,  X over  the  first  5 years.  This  program  struc- 
ture is  applicable  only  for  evolutionary  ship  de- 
signs with  long  construction  times,  i.e.,  typical 
displacement-hull  warships. 

The  prototype  ship  acquisition  program  is 
one  that  includes  significant  research  and  de- 
velopment funding  for  a prototype,  usually  a 
new  design  incorporating  major  technical  ad- 
vancements not  proved  in  nonnuclear  propul- 
sion of  hull  design.  Examples  of  such  innovative 
and  unique  ship  types  are  hydrofoils,  air  cush- 
ion vehicles,  surface  effect  and  small  water- 
plane-area  twin-hull  ships.  In  these  and  other 
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advanced  concepts  in  hull  form  and  propulsion 
the  Navy  hopes  for  a quantum  improvement  in 
performance  across  the  surface  of  the  ocean  (in- 
stead of  plowing  through  the  sea  with  a dis- 
placement hull).  In  this  type  of  ship  acquisition 
program,  the  unknowns  are  many  and  the  is- 
sues go  beyond  a particular  design.  The  entire 
operational  concept  and  the  military  potential 
of  the  concept  are  scrutinized.  The  principal 
means  for  such  a determination  is  initial  opera- 
tional test  and  evaluation  of  the  prototype  war- 
ship at  sea  before  Milestone  III,  the  first  major 
production  decision  for  follow-ship  acquisition. 

Usually  the  prototype  with  weaponry  will 
undergo  a full  ship  operational  evaluation  that 
evaluates  the  ship,  not  specific  equipments.  The 
individual  equipments  will  have  been  subjected 
to  prior  operational  evaluation  within  individu- 
al development  programs.  The  ship  OPEVAL 
objective  is  to  see  how  well  the  ship  performs 
her  missions,  not  how  well  a particular  system 
works.  Indeed,  the  prototype  ship  operational 
evaluation  goes  further  and  challenges  the  need 
for  any  planned  follow-on  acquisition.  For  ex- 
ample, an  operational  evaluation  was  con- 
ducted at  sea  by  Commander,  Operational  Test 
and  Evaluation  Force  on  the  NATO  Patrol 
Missile  Hydrofoil.  From  this  testing  not  only 
the  operational  effectiveness  and  operational 
suitability  of  the  PHM  advanced-design  pro- 
pulsion and  control  systems,  struts,  and  foils 
were  evaluated,  but  also  evaluated  was  the  op- 
erational concept  of  a high  speed,  missile-firing 
fleet  ship.* 

The  OPEVAL,  a term  unique  to  the  Navy, 
is  the  final  phase  of  a system’s  initial  opera- 
tional test  and  evaluation  (OT-III),  and  takes 
place  immediately  before  the  first  major 
production  decision.  The  test  period  is  called 
OT-III  to  indicate  that  it  precedes  Milestone 


•In  the  10  years  before  PHM,  the  Navy  designed, 
constructed,  and  operated  four  hydrofoil  ships  (USS 
HIGH  POINT,  USS  PLAINVIEW,  USS  TUCUM- 
CARI,  USS  FLAGSTAFF).  The  PHM  class  is  the  first 
USN  hydrofoil  constructed  as  a fleet  warship  with  ade- 
quate firepower  and  command,  control,  and  communica- 
tions capability  to  permit  operational  risk,  as  well  as 
technical  risk,  to  be  addressed  thoroughly  through  test 
and  evaluation. 
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III.  The  OPEVAL  always  requires  operation 
and  maintenance  by  fleet-type  user  personnel 
with  the  test  system  in  a configuration  as  near 
the  production  configuration  as  possible.  The 
evaluation  is  usually  structured  to  stress  the 
system  in  scenario-type  tests,  at  sea,  with  all  the 
vagaries  of  weather,  threat,  and  operators. 

Less  visible  than  OPEVAL  is  early  initial 
operational  test  and  evaluation,  OT-I  and  OT- 
11,  that  occur  before  acquisition  Milestones  I 
and  II,  respectively.  The  tests  before  Milestone 
I are  usually  addressed  to  concepts  and  feasibil- 
ity. The  extensive  at-sea  tests  in  the  early  Seven- 
ties of  a conceptual  sea  control  ship  is  an  exam- 
ple of  OT-I.  In  these  tests,  the  USS  GUAM 
(LPH  9)  was  designated  an  interim  sea  control 
ship  for  test  puiposes  and  new  antisubmarine 
warfare  task  force  tactics  were  evaluated.  The 
tests  before  Milestone  II  represent  the  more 
critical  early  initial  operational  test  and  evalua- 
tion period  because  it  holds  the  greatest  poten- 
tial to  influence  future  fleet  equipment  from  the 
operational  point-of-view.  These  tests  must  use 
fleet-type  personnel  for  hands-on  operation  of 
the  equipment,  but  may  use  developing 
agency/contractor  personnel  for  maintenance 
and  support.  Pre-Milestone  II  testing  is  con- 
cerned with  the  operational  direction  the 
weapon  system  design  is  taking,  and  its  poten- 
tial to  be  operationally  effective  and  operation- 
ally suitable  in  the  fleet.  Follow-on  operational 
test  and  evaluation  (OT-IV  and  OT-V)  occurs 
after  Milestone  III,  and  is  directed  to  actual 
demonstration  of  operational  performance  and 
tactics.  The  difference  is  primarily  that 
production  systems  are  tested,  and  in  OT-V,  all 
resources  are  budgeted  by  the  Fleet  Com- 
mander instead  of  the  acquisition  manager. 

INITIAL  OPERATIONAL  TEST 
AND  EVALUATION  AND  RISK 
REDUCTION 

Initial  operational  test  and  evaluation  has 
as  its  objective  an  actual  demonstration  that  op- 
erational risks  have  been  reduced  to  an  accept- 
able level  before  the  system  is  placed  in  serial 
production.  On  a lesser  system,  one  not  so  com- 
plex as  a ship,  OPEVAL  would  be  conducted 
on  a mature  or  nearly  mature  design,  with  the 


expectation  that  the  production  unit  would  not 
differ  significantly  from  the  model  tested.  Typi- 
cal examples  include  a new  shipboard  radar, 
propulsion  engine,  or  fire  control  system.  In 
conventional  warship  acquisition,  the  situation 
IS  quite  different.  Since  the  ship  does  not  exist  at 
Milestone  III  or  IIIA,  initial  operational  test 
and  evaluation  must  be  suboptimized  by  look- 
ing at  parts  of  the  eventual  total  ship  and  esti- 
mating potential  operational  effectiveness  and 
suitability.  Consequently,  the  decisionniaker  in 
reviewing  the  results  of  initial  operational  test 
and  evaluation  is  concerned  less  with  the  details 
of  design,  and  more  with  the  direction  the  de- 
sign is  taking  and  the  track  record  of  the  devel- 
oper \n  meeting  the  specific  milestone  technical 
and  operational  performance  objectives.  The 
“earlier-than-usual”  decisions  required  in  ship- 
building programs  are  inherently  risky.  In  addi- 
tion to  increased  risk,  the  probability  increases 
that  the  design  or  operational  concept  will  in- 
clude not  only  the  known-unknowns,  but  also 
disquieting  unknown-unknowns — surprised 

The  decisionmaker  is  interested  in  all 
risks — technical  and  operational  performance, 
cost,  and  schedule.  Initial  operational  test  and 
evaluation  is  concentrated  only  on  demon- 
strated operational  performance.  The  concept 
of  technical  risk  is  generally  well  understood, 
and  is  assessed  by  addressing  the  questions: 

• Does  the  system  meet  the  required 
technical  characteristics  and  specifica- 
tions? 

• If  not,  what  is  yet  to  be  accomplished? 

• Is  the  remaining  engineering  develop- 
ment within  the  state-of-the-art? 

• Can  the  system  be  produced  by  indus- 
try? 

The  concept  of  operational  risk,  while  not 
nearly  so  well  understood,  is  receiving  in- 
crease attention.  Operational  risk  is  the  risk 
that  the  warship  might  not  be  militarily  useful 
once  put  to  sea,  achievement  of  stated  require- 
ments or  specifications  notwithstanding.  This 
could  be  the  result  of  the  wrong  specifications 
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or  technology  applications  having  been  in- 
voked, lack  of  knowledge  or  training  in  the  tac- 
tical use  of  the  ship,  or  perhaps  an  inadequate 
assessment  or  oversimplified  assumptions 
about  the  environment  in  which  the  ship  would 
operate.  Failure  could  result  from  the  un- 
known-unknowns in  areas  never  addressed  in 
the  design  or  requirements  process.  Considera- 
tion of  these  operational  risks  is  of  overriding 
importance,  particularly  in  development  of  an 
entirely  new  ship-type  like  a hydrofoil  or  sur- 
face effect  ship,  because  the  ship  will  not  be  mil- 
itarily useful  without  development  of  support- 
ing systems,  nor  will  it  be  useful  without  tactics 
and  procedures  for  use  in  combat  under  nonbe- 
nign  conditions. 

These  operational  risks,  to  a greater  or 
lesser  degree,  exist  in  all  shipbuilding  programs, 
but  particularly  in  advanced  technology  areas, 
for  example,  when  prototyping  is  necessary. 
The  initial  operational  test  and  evaluation  nec- 
essary to  highlight  the  degree  of  achievement  of 
performance  objectives  relative  to  the  remain- 
ing operational  risk,  has  become  a fact  of  life 
taking  several  forms. 

Ship  Design 

For  initial  operational  test  and  evaluation 
during  the  ship  design  phase,  operational  evalu- 
ation plays  a far  greater  role  than  operational 
testing.  The  role  of  the  Commander,  Opera- 
tional Test  and  Evaluation  Force  is  to  continu- 
ously evaluate  the  design  as  it  matures,  and 
carry  the  fleet  operator’s  message  to  the  design 
engineer.  The  operational  test  director  does  not 
drive  the  design  in  any  way,  but  rather  is  avail- 
able to  transfer  firsthand  knowledge  and  at-sea 
experience  to  assist  the  designer  in  understand- 
ing how  the  ship  and  its  equipments  will  be 
used.  Through  participation  in  design  reviews, 
and  evaluation  of  compartment  arrangements, 
equipment  mock-ups,  and  the  output  of  com- 
puter simulation  models  (or  any  valid  design 
and  engineering  tools),  the  operational  test  di- 
rector draws  on  his  skills,  experience,  and  abil- 
ity to  conceptualize  ultimate  fleet  employment 
of  the  ship  to  make  analyses  and  draw  conclu- 
sions. 


The  discourse  is  two-way.  It  is  not,  during 
the  ship  design  and  engineering  phase,  a simple 
evaluator/evaluated  relationship.  The  opera- 
tional test  director  attempts,  during  the  re- 
views, to  gain  a fuller  understanding  of  the  de- 
sign. This  understanding  cannot  come  from  a 
reading  of  specifications  and  drawings  alone.  It 
is  equally  important  that  the  operational  test  di- 
rector hear  the  specification  and  employment 
concept,  and  learn  exactly  what  it  is  that  the  de- 
signer is  trying  to  accomplish,  and  what  as- 
sumptions are  implicit  but  perhaps  not  obvious 
in  the  design.  With  this  understanding,  the  op- 
erational test  director  can  proceed  to  construct 
meaningful  and  realistic  tests — tests  that  will 
allow  the  ship  design  to  prove  itself  capable  of 
meeting  the  letter  and  spirit  of  the  original  ship 
acquisition  program  operational  requirements 
as  approved  by  the  Chief  of  Naval  Operations. 

When  reviewing  the  ship  design  and  for- 
mulating test  plans,  the  operational  test  direc- 
tor is  able  to  challenge  the  design.  He  can  ask 
“what  if’  questions  to  define  operational  risks 
clearly  and  to  surface  operational  issues  that 
might  need  further  attention.  It  is  this  evalua- 
tion, rather  than  test,  that  constitutes  the 
important  portion  of  initial  operational  test  and 
evaluation  in  the  ship  design  phase. 

Individual  Systems 

Parallel  to  the  ship  design  phase,  individ- 
ual subsystems  identified  for  incorporation  into 
the  new  warship  (guns,  radars,  engines,  aircraft 
catapults,  missiles,  etc.)  undergo  separate  initial 
operational  test  and  evaluation  within  the  indi- 
vidual development  programs.  This  evaluation 
culminates  in  individual  OPEVALS  and  Mile- 
stones III,  prior  to  the  ship  acquisition  Mile- 
stone III.  The  final  phase  OPEVAL  usually  is 
the  most  rigorous  phase  of  initial  operational 
test  and  evaluation,  planned  and  conducted  by 
COMOPTEVFOR  using  fleet  personnel  at  sea 
in  threat-oriented  scenarios.  The  test  results, 
properly  evaluated,  can  affect  not  only  the  fu- 
ture of  the  subsystem,  but  also  the  ship  for 
which  the  system  is  intended.  The  Commander, 
Operational  Test  and  Evaluation  Force’s  report 
of  an  OPEV AL  to  the  Chief  of  Naval  Opera- 
tions includes  an  assessment  of  the  weapon  sys- 
tem’s operational  effectiveness  and  operational 
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suitability,  together  with  an  approval  for  ser- 
vice use  recommendation.  If  a subsystem  repre- 
sents a significant  capability  within  the  ship,  its 
OPEVAL  outcome  could  become  an  issue  in 
the  ship  production  decision. 

Land  Based  Testing 

Given  that  ship  design  and  individual  sub- 
systems have  been  properly  evaluated,  the  next 
step  in  ship  acquisition  initial  operational  test 
and  evaluation  is  to  assess  the  amalgamation  of 
individual  equipments  into  a higher  level  com- 
bat system  or  propulsion  system  at  a land  based 
test  site.  Land  based  test  sites  are  not  the  rule. 
When  complexity  or  technical  advancement 
warrants,  a LETS  is  constructed.  The  site  is  to 
duplicate  and  simulate  as  many  elements  of  the 
selected  subsystem’s  planned  installation  as 
practicable,  and  necessary,  to  reduce  acquisi- 
tion program  risk  before  Milestone  IIIA. 


As  an  example  of  how  a land  based  test  site 
might  be  used,  consider  the  propulsion  system 
for  a new  warship  class.  The  propulsion  system 
test  site  ashore  is  usually  incomplete  in  that  it 
lacks  the  ship’s  hull  and  propeller.  What  usu- 
ally is  installed  is  the  power  train,  from  prime 
mover  through  propeller  shafting,  and  the  con- 
trol system.  A water  brake  or  other  device 
serves  as  a load  on  the  system  to  simulate  ocean 
action  on  the  ship  and  propeller.  With  these 
limitations  to  a performance  evaluation,  initial 
operational  test  and  evaluation  leans  heavily  to- 
ward assessing  the  adequacy  of  component  in- 
tegration and  evaluation  of  operational  suita- 
bility issues  such  as  reliability,  maintainability, 
and  compatibility,  where  early  problem  detec- 
tion can  result  in  significant  cost  savings.  While 
effectiveness  issues  do  exist  in  the  prime  mover 
and  propulsion  train,  these  issues  are  for  the 
most  part  technical  in  nature,  such  as  horse- 
power developed  or  fuel  consumption  rates. 
The  real  operational  effectiveness  issue  for  new 
propulsion  systems  is  how  well  the  control  sys- 
tem performs  when  it  is  required  to  orchestrate 
other  propulsion  system  elements  during  re- 
peated maneuvers  matched  to  the  operating 
profile  of  a warship  at  sea. 


Although  not  present  at  the  land  based  test 
site,  the  propeller  will,  when  appropriate,  un- 
dergo separate  at-sea  operational  lest  and  eval- 
uation on  a surrogate  ship.  To  support  use  of 
gas  turbine  propulsion  in  the  DD  963  and  FFG 
7 classes,  it  was  necessary  for  the  Navy  to  de- 
velop a new  propeller  capable  of  transmitting 
thrust  in  the  ^,000  shaft  horsepower  range,  yet 
capable  of  reversing  pitch  (gas  turbines  are 
unidirectional).  Two  competing  designs  were 
put  to  sea  in  USS  BARBEY  (FF  1088)  and  USS 
PATTERSON  (FF  1061)  for  combined  devel- 
opment and  operational  test  and  evaluation. 
The  failures  that  occurred  during  this  com- 
bined test  and  evaluation  led  to  additional  re- 
search and  development  in  propeller  design  and 
control.  Further  engineering  ultimately  permit- 
ted the  introduction  of  gas  turbine  propulsion 
to  the  fleet.* 

Despite  propulsion  train  testing  at  the  land 
based  test  site  and  propeller  testing  at  sea,  sev- 
eral parts  of  the  propulsion/mobility  system  are 
neither  integrated  nor  tested  prior  to  construc- 
tion of  the  lead  ship.  The  land  based  test  site 
may  not  include,  or  exactly  duplicate,  auxiliary 
and  support  systems  directly  related  to  the  pro- 
pulsion train  as  it  will  be  installed  in  the  ship. 
These  support  systems  might  include;  lube  oil, 
feed  water,  electrical,  and  compressed  air.  Such 
systems  are  not  easily  defined  or  bounded  and 
often  do  not  go  through  the  standard  opera- 
tional test  and  evaluation/approval  for  service 
use  process.  These  are  systems  of  the  “as  re- 
quired’’ type,  usually  contractor-furnished,  and 
unique  to  each  ship  class.  While  these  systems 
escape  initial  operational  test  and  evaluation  at 
the  land  based  test  site,  they  cannot  be  over- 
looked. Failure  of  any  of  these  systems  can  re- 
sult in  loss  of  warship  mobility.  The  Navy  re- 
quires that  these  systems  meet  the  objectives  of 
the  approval  for  service  use  process,  even 
though  furnished  by  the  shipbuilder  as  an  inte- 
gral part  of  the  shipbuilding  contract.’ 

Department  of  Defense  Directive  5(X)0.3  is 
specific  about  the  prototyping  of  new  propul- 
sion systems,  although  a specific  statement  is 
not  included  that  requires  a propulsion  system 
land-based  test  site.  The  Navy  addresses  this 
need  in  Navy  directives.*  Recent  experience  in 


the  NATO  patrol  missile  hydrofoil  program, 
which  had  both  a propulsion  land  based  test  site 
and  a prototype  ship,  suggests  that  where  an 
RDT&E,N-funded*  prototype  ship  exists,  it 
may  be  more  cost-effective  to  use  the  prototype 
ship  for  propulsion  system  testing  rather  than  a 
land  based  test  site.  In  the  patrol  missile  hydro- 
foil case,  all  operational  test  and  evaluation 
data  were  collected  during  prototype  testing  at 
sea  in  1976.  Operational  test  and  evaluation 
was  not  conducted  at  the  land  based  test  site.’ 

The  combat  system  test  site  ashore  differs 
significantly  from  the  propulsion  land  based 
test  site.  While  also  lacking  the  ship  and  ocean 
environment,  it  can  be  used  for  operational  per- 
formance evaluation  through  the  use  of  compu- 
ter simulation  methods.  Installed  at  the  combat 
system  land  based  test  site  are  a combination  of 
warship  equipments  and  simulators.  These 
items  represent  the  ship’s  sensor,  weapon,  and 
information  processing  subsystems  integrated 
through  interfaces,  e.g.,  digital  computer  tech- 
niques. The  site  must  include  a physical  and 
functional  replication  of  the  ship’s  combat  in- 
formation center  if  it  is  to  be  of  significant  value 
for  initial  operational  test  and  evaluation.  Addi- 
tionally, there  will  be  some  type  of  test  control 
center  from  which  the  COMOPTEVFOR  oper- 
ational test  director  and  his  evaluation  team 
can  stimulate  the  ship’s  combat  information 
center  with  a wartime  scenario  by  introducing 
targets,  communications,  and  bits  of  intelli- 
gence to  which  the  combat  system  must  react. 
Thus  much  of  the  combat  system  capability  can 
be  evaluated  including  equipment,  computer 
programs,  and  manning. 

The  goal  of  combat  system  land-based  op- 
erational test  and  evaluation  is  an  evaluation  of 
the  intended  ship’s  man/machine  interface. 
Combat  information  center  is  simply  the  loca- 
tion where  the  man/machine  interface  takes 
place;  and  “combat  system  integration” — 
which  is  tested  at  the  site — is  simply  the  means 
by  which  it  is  done.  This  man/machine  inter- 
face is  the  central  focus  of  initial  operational 


•Research  Development  Test  and  Evaluation  money 
appropriated  for  Navy  u.se. 


test  and  evaluation  at  the  combat  system  land- 
based  test  site.  On  the  “man”  side  of  the  equa- 
tion, the  goal  of  the  test  site  is  achieved  by; 

• deciding  upon  the  stationing  of  person- 
nel; 

• deciding  upon  radio/interior  commu- 
nications/voice communication  pat- 
terns; 

• developing  a “combat  system  doc- 
trine”; and 

• training. 

The  “man”  side  of  the  equation  can  be  ad- 
justed after  ship  construction. 

On  the  “machine”  side  of  the  equation,  the 
goal  is  achieved  by  numbers,  location,  capabil- 
ity, and  interconnection  of  equipments.  The 
“machine”  side  (whether  equipment  or  com- 
puter programs)  is  relatively  expensive  to 
change  after  ship  construction.  Thus  a goal  for 
operational  testing  at  the  combat  system  test 
site  is  to  bring  the  “machine”  side  of  the  inter- 
face to  an  adequate  capability  level  before  ship 
construction.  In  this  way  subsequent  post- 
construction fine-tuning  can  be  done  on  the 
“man”  side  of  the  equation. 

Initial  operational  test  and  evaluation  at 
the  FFG  7 combat  system  land  based  test  site 
resulted  in  major  equipment  and  configuration 
changes  in  the  combat  information  center  to 
make  the  design  operationally  effective  for  one 
of  the  ship’s  major  mission  areas. 

The  combat  system  land  based  test  site  is 
not  without  limitations.  Some  key  system  ele- 
ments cannot  function  at  a shore  site.  Sonars, 
underwater  listening  devices,  towed  arrays, 
missile,  gun,  torpedo,  and  chaff  firings  must  be 
simulated.  Likewise,  surface  search  radars,  ac- 
tive electronic  countermeasures,  navigation 
systems,  etc.,  cannot  or  are  not  allowed  to  be 
operated  ashore  in  the  same  manner  as  at  sea. 
Outside  of  the  combat  information  center  and 
other  replicated  operating  spaces,  the  support 
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systems,  such  as  electrical  power,  cooling 
water,  and  compressed  gas  systems,  usually 
bear  little  resemblance  to  the  intended  ship 
configuration. 

The  emphasis  here  on  the  shortcomings  of 
land-based  testing  is  not  intended  to  degrade 
the  importance  of  these  sites.  Land  based  sites 
serve  as  a valuable  engineering  tool  in  the  early 
reduction  of  risks,  technical  and  operational. 
The  intent  is  to  point  out  the  need  for  opera- 
tional testing  at  sea.  The  combat  system  and 
propulsion  system  are  not  two  independent 
blocks  that  can  be  stacked  one  above  the  other 
on  a hull  without  further  integration,  and  then 
be  put  to  sea  as  a warship.  Ship  integration,  pro- 
pulsion or  combat  system,  is  not  accomplished 
by  summing  the  individual  elements.  T/ie  ele- 
ments are  not  independent.  The  elements  can 
complement  each  other  and  they  can  interfere 
with  each  other.  At  sea,  system  integration  with 
the  hull  is  stressed  in  operational  testing  and  at- 
tempts are  made  to  measure  system  perfor- 
mance, at  least  in  qualitative  terms.  This  ship 
integration  is  something  that  cannot  be  repro- 
duced ashore  at  a test  site. 

The  actual  warship  capability  may  be 
more  or  less  than  the  sum  of  the  capabilities  of 
all  the  elements.  We  do  look  for  something  ex- 
tra as  a result  of  the  integration.  Perhaps  it  is  a 
faster  reaction  time  or  fineness  of  control,  but 
whatever  we  seek  we  find  that  a major  portion 
of  the  integration  effort  cannot  be  accom- 
plished during  the  ship  design  phase  and  must, 
of  necessity,  be  undertaken  concurrent  with  the 
lead  ship’s  construction  and  fleet  introduction. 
To  make  this  apparent  concurrence  in  design 
and  acquisition  work  productively,  the  Ship 
Acquisition  Program  Manager  must  include  in- 
itial operational  test  and  evaluation  as  an  im- 
portant tool  in  pacing  program  progress. 

A complete  understanding  of  the  purpose 
of  any  test  site  is  essential.  The  land-based  test 
site  must  allow  not  only  the  design  and  engi- 
neering effort,  but  also  the  test  and  evaluation 
to  demonstrate  that  the  integration  has  been 
satisfactorily  achieved.  Inherent  in  the  capabil- 
ity to  simulate  portions  of  the  system  is  the  pit- 
fall  of  unrealistic  simulation.  Accordingly,  re- 
sults of  land-based  test  site  initial  operational 


test  and  evaluation,  and  particularly  combat 
system  initial  operational  test  and  evaluation, 
are  heavily  dependent  on  the  validity  with 
which  the  behavior  of  the  missing  equipment, 
ship,  and  its  external  environment  (threat  and 
natural)  are  replicated  and  perceived  by  opera- 
tors participating  in  the  tests. 

A given  combat  system  land  based  test  site 
may  provide  adequate  simulation  for  the  de- 
signer and  engineer  to  demonstrate  that  the 
ship’s  combat  system  functions  as  specified,  in  a 
technically  acceptable  manner,  and  that  the 
known-unknown  risks  have  been  reduced.  This 
is  usually  demonstrated  by  playing  a pre- 
programmed scenario  to  stimulate  the  combat 
information  center.  The  ship’s  crew  responds  to 
the  air,  surface,  or  subsurface  threats.  The 
threats  are  presented  so  that  tactical  decision- 
making is  simplistic  and  sequential.  Own  ship 
combat  information  center  is,  normally,  the  vic- 
tor. Engagements  are  not  constrained  by  realis- 
tic limitations  in  the  ship’s  weapon  load  as  to 
type  and  quantity,  fuel  remaining  on  board, 
heavy  weather,  radio  communications  and  in- 
telligence failures,  etc.  Operator  frailties,  par- 
ticularly under  stress,  are  discounted  as  notests. 
If  meaningful  initial  operational  test  and  evalu- 
ation is  to  be  conducted  at  this  combat  system 
land  based  test  site,  additional  expenditures  are 
necessary  to  simulate  an  enemy  that  fights 
back,  i.e.,  that  can  be  maneuvered  in  response 
to  own  ship  combat  information  center  actions. 

Operational  test  and  evaluation  is  con- 
cerned with  the  interoperability  of  the  ship 
combat  information  center  during  command, 
control,  and  communications  interfaces  with 
other  ships  and  aircraft  in  the  simulated  task 
force.  Tests  highlight  the  ship's  combat  infor- 
mation center  arrangement,  equipment,  opera- 
tor tools,  and  interconnections — all  things  that 
can  be  changed  before  the  ship  is  actually  con- 
structed. These  items  are  costly  to  change  once 
installed.  What  is  not  of  concern  in  this  opera- 
tional testing  is  who  wins  or  loses  the  simulated 
combat  engagement  because  that  was  addressed 
before  the  extensive  Navy  commitment  to  a test 
site  and  a shipbuilding  program.  The  question 
to  be  answered  by  this  operational  testing  is 
what  must  be  modified  to  increase  the  warship's 
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operational  effectiveness  and  operational  suita- 
bility? From  operational  test  and  evaluation 
should  come  identification  of  some  unknown- 
unknowns,  and  reduction  of  operational  risk. 

All  types  of  initial  operational  test  and 
evaluation  have  had  as  an  objective  the  demon- 
stration of  program  objective  achievement  by 
Milestone  IIIA.  At  this  point  the  key  decision  is 
made  to  commit  dollars  to  follow-ship  produc- 
tion, that  is,  award  of  construction  contract. 
Test  and  evaluation,  both  operational  and  tech- 
nical, must  demonstrate  system  capabilities  suf- 
ficient to  give  the  decisionmaker  a clear  under- 
standing of  the  risk  involved.  It  is  not  necessary 
that  all  risk  be  eliminated.  It  is  necessary  that 
risk  be  reduced  to  an  acceptable  and  manage- 
able level,  given  the  planned  shipbuilding 
schedule;  and  where  risk  still  remains,  that 
there  exist  a firm  plan  for  its  resolution,  to  in- 
clude a fallback  position  should  resolution  fail. 

FOLLOW-ON  TEST  AND 
EVALUATION 

The  culmination  of  initial  operational  test 
and  evaluation  at  the  land  based  test  site  and 
subsystem  OPEVALS  occurs  immediately  be- 
fore the  Milestone  IIIA  decision.  Deficiencies 
are  identified,  and  fixes  are  developed.  The  ship 
contracting  process  can  proceed  after  Milestone 
IIIA  while  land  based  test  site  and  OPEVAL 
deficiencies  undergo  an  interactive  process  of 
fix  and  retest  until  equipment  and  systems  are 
operationally  effective  and  suitable  to  receive 
unconditional  approval  for  service  use. 

After  the  ship  Milestone  IIIA  decision  and 
contract  award  for  the  first  follow-ship,  at  least 
2 years  remain  before  the  lead  ship  is  delivered 
and  put  to  se  *•  This  time  is  not  lost,  because  im- 
plicit in  Navy  ship  acquisition  program  struc- 
ture and  supporting  test  and  evaluation  policy 
is  the  knowledge  that  engineering  and  testing 
will  continue.  It  is  partially  in  recognition  of 
this  planned  test  and  evaluation  for  risk  reduc- 
tion, as  documented  in  the  TEMP,  that  the 
follow-ship  decision  can  be  made  well  before 
the  lead  ship  is  launched. 

The  long  design  and  construction  time  for 
a conventional  warship  permits  the  Navy  to 


capitalize  on  the  opportunity  to  make  changes 
in  equipment  design,  operating  compartment 
arrangement,  and  computer  program  configu- 
ration, as  well  as  adjustments  to  manning.  The 
changes  and  adjustments  are  further  tested  at 
sea  on  surrogate  ships  and  ashore  at  the  land 
based  test  site.  The  Navy’s  newest  class  of  frig- 
ates (FFG  7),  for  example,  passed  Milestone 
IIIA  in  late  1975.  Based  on  initial  operational 
test  and  evaluation  results,  the  combat  informa- 
tion center  was  almost  completely  rearranged, 
new  equipment  added,  and  computer  programs 
modified.  Eighteen  months  later,  this  rede- 
signed combat  system  was  subjected  to  follow- 
on  operational  test  and  evaluation  at  the  land 
based  test  site  by  COMOPTEVFOR,  several 
months  before  the  lead  shipv/ns  scheduled  to  be 
delivered.  Similar  follow-on  operational  test 
and  evaluation  was  conducted  at  the  propulsion 
system  land  based  test  site. 

Delivery  of  the  lead  ship  by  the  builder  is 
followed  by  rigorous  Acceptance  Trials  by  the 
President,  Board  of  Inspection  and  Survey. 
This  is  followed  by  extensive  tests,  trials,  and 
training  under  direction  of  the  Chief  of  Naval 
Material  and  the  Fleet  Commander.  Next,  se- 
lected ship  subsystems  and  equipments  undergo 
further  operational  test  and  evaluation  to  verify 
correction  of  remaining  OPEVAL  deficiencies, 
complete  the  ASU  process  where  necessary, 
and  complete  system  integration.  For  both  the 
propulsion  and  combat  systems,  the  lead  ship  is 
really  the  first  time  that  everything  is  integrated 
completely,  placed  in  the  at-sea  environment 
with  a trained  crew,  and  tested  in  mission-and- 
threat-representative  scenarios  with  live 
weapon  firings. 

The  usual  fleet  introduction  period  for  a 
new  warship  is  about  1 1 months,  during  which 
time  tests,  trials,  training,  and  a shipyard  avail- 
ability occur.  Squeezed  into  this  period  are  3 to 
6 weeks  of  follow-on  test  and  evaluation  by 
COMOPTEVFOR  on  propulsion  and  combat 
system  elements  (where  necessary  to  complete 
the  approval  for  service  use  proce.ss),  and  on  the 
integration  of  these  elements.  Lead-shtf  test 
and  evaluation  concentrates  on  the  remaining 
operational  effectiveness  and  operational  suita- 
bility issues. 
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WHOLE-SHIP  OPERATIONAL  • Individual  system  technical  per- 

TEST  AND  EVALUATION  formance. 


In  addition  to  the  continuing  phases  of  op- 
erational test  and  evaluation  on  the  lead  ship  of 
a new  class  at  sea,  the  operational  test  and  eval- 
uation policy  of  DOD  Directive  5000.3  in- 
cludes a phrase  that  allows  “...if  required,  full 
ship  operational  evaluation  to  the  degree  feasi- 
ble.” This  is  a new  concept  to  the  Navy.  Notice 
that  this  policy  permits  the  decisionmaker  con- 
siderable latitude  for  defining  need  and  scope  of 
“operational  test  and  evaluation  of  a ship  as  a 
whole.” 

The  Navy  has  studied  this  new  concept  in 
great  detail  for  two  ship  classes,  USS  VIRGIN- 
IA (CGN  38)  and  USS  OLIVER  H.  PERRY 
(FFG  7),  both  typical  displacement-hull  war- 
ships. The  concept  developed  by 
COMOPTEVFOR  was  to  evaluate  a ship,  not 
specific  equipments.  The  individual  equipments 
in  the  ship  would  have  been  evaluated  previous- 
ly, singly  and  as  elements  of  an  integrated  com- 
bat or  propulsion  system. 

NOTE 

The  objective  of  doing  operational 
test  and  evaluation  on  an  entire  ship 
is  to  evaluate  how  well  the  ship  per- 
forms her  missions,  not  how  well  a 
particular  system  works.  Since  a 
ship’s  mission  necessitates  accom- 
plishment of  a wide  variety  of  tasks 
simultaneously  in  a combat  environ- 
ment, the  operational  test  and  evalua- 
tion must  take  place  with  the  ship  in 
an  operational  environment. 

The  approach  to  whole-ship  opera- 
tional test  and  evaluation  in  a lead- 
ship,  follow-ship  acquisition  program 
in  this  concept  is  two-phased.  Phase  I 
is  the  preliminary  phase,  to  provide 
an  understanding  of  the  ship’s  perfor- 
mance with  normal  manning,  under 
fleet  control,  in  an  operational 
environment.  Phase  I would  be  con- 
ducted sequentially  through  a series 
of  tests  and  evaluations  to  determine: 


• Technical  capability  of  systems 
in  integrated  operation. 

• System  operational  effectiveness 
afloat. 

• System  operational  suitability 
afloat. 

Phase  II,  the  operational  phase, 
would  determine  whole-ship  opera- 
tional effectiveness  and  operational 
suitability  in  a threat  environment. 
This  phase  would  ensure  that: 

• All  significant  aspects  of  the 
ship’s  mission  performance  are 
evaluated. 

• Any  potential  problem  areas  (e.g., 
systems  interfaces)  have  been 
identified,  and  the  problem  im- 
pact on  mission  performance  has 
been  assessed. 

To  avoid  duplication  of  testing  be- 
tween phases,  every  effort  would  be 
made  to  conduct  Phase  I tests  on  the 
lead  ship  under  conditions  such  that 
repetitive  Phase  II  tests  could  be 
avoided.  Phase  II  would  cover  only 
that  essential  test  and  evaluation  not 
previously  accomplished. 

The  preceding  discussion  cf  tin  con- 
cept deals  with  a short-range  evalua- 
tion of  the  ship,  i.e.,  how  effective  is 
she  with  current  sensors,  weapons, 
and  computer  programs?  There  is  an- 
other important  aspect  of  a new  war- 
ship, and  that  is  her  long-range  eitec- 
tiveness.  How  well  will  the  ship  do 
over  a 30-year  life?  This  type  of  eval- 
uation involves  growth  potential, 
ease  of  update,  modularity,  space  and 
weight  reservations,  excess  power 
and  water,  excess  crew  space,  etc.,  as 


well  as  durability  and  maintainabil- 
ity of  the  basic  ship  systems.  While 
Phase  II  operational  testing  would 
concentrate  on  short-range  aspects  of 
the  ship,  COMOPTEVFOR  would 
provide  both  a short-range  and  a 
long-range  assessment. 

As  a result  of  Milestone  II  for  the  FFG  7 
(formerly  Patrol  Frigate  PF)  program  in  1972, 
the  Deputy  Secretary  of  Defense  stated  the  gen- 
eral purpose  of  whole-ship  operational  test  and 
evaluation  in  the  decision  memorandum  for 
Milestone  II. 

“...Also,  it  may  be  desirable  that  a pe- 
riod for  operational  test  and  evaluation 
of  the  lead  ship,  prior  to  that  ship’s  full 
release  to  normal  Fleet  usage,  be  allo- 
cated to  OPTEVFOR.  The  purpose  of 
this  testing  would  be  to  determine  the 
PF’s  expected  operational  effectiveness 
in  its  expected  roles  and  the  need  for 
any  early  modification  to  follow  ships. 
Should  such  modifications  be  required, 
a later  DSARC  would  have  to  deter- 
mine the  relative  merits  of  opening  ex- 
isting contracts  to  change  by  change 
order  procedures  or  making  modifica- 
tions after  acceptance  from  the  ship- 
builder.’’* 

In  simple  terms,  the  reasons  for  doing 
whole-ship  operational  test  and  evaluation  are: 

1 . to  find  out,  at  the  earliest  possible  time, 
what  the  ship  capabilities  in  mission  terms  re- 
ally are, 

2.  to  discover  any  weaknesses  the  ship  may 
have — again,  as  soon  as  possible — so  we  can  ei- 
ther correct  them  or  learn  how  to  live  with 
them, 

3.  to  confirm  correction  of  discrepancies 
noted  on  previous  trials/tests, 

4.  to  identify  key  logistics  considerations 
(e.g.,  repair  parts,  maintenance  support,  train- 
ing) which  should  be  modified  or  established  as 


soon  as  possible  to  ensure  that  the  fleet  can  op- 
erate, maintain,  and  support  the  ship, 

5.  to  provide  early  feedback  to 
OPNAV/NAVMAT,  to  assist  in  work  on  fu- 
ture ships  and  systems,  and 

6.  to  develop  initial  procedures/tactics 
that  specify  how  to  best  employ  the  ship. 

These  early  insights  on  tactical  employment  are 
a by-product  of  the  operational  appraisal — 
turned  over  to  Fleet  Commanders  with  the 
ship,  for  further  development. 

The  usual  reason  for  conducting  an 
OPEVAL — to  verify  that  the  system  is  ready 
for  production — does  not  apply  in  the  case  of  a 
conventional,  displacement-hull  ship.  By  the 
time  the  lead  ship  has  completed  an  operational 
appraisal,  all  other  ships  of  the  class  should 
have  been  contracted  for  and  several  will  be 
well  along  in  construction. 

The  above  discussion  has  addressed  whole- 
ship  operational  test  and  evaluation  using  a typ- 
ical displacement-hull  warship,  after  all  lower 
levels  of  test  and  evaluation  are  complete — a 
process  that  may  take  many  months,  even 
years.  Contrast  this  with  the  prototype  ship  ac- 
quisition program  where  a ship  OPEVAL  is 
conducted  be/bre Milestone  III. 

The  decision  for  both  acquisition  pro- 
grams studied  to  date  by  the  Navy  (USS  VIR- 
GINIA and  USS  OLIVER  H.  PERRY)  was 
not  to  require  operational  evaluation  of  the  lead 
ship  (whole-ship  operational  test  and 
evaluation).  Given  the  policy  in  DOD  Directive 
50(XI.3,  it  can  be  expected  that  the  requirement 
for  whole-ship  operational  test  and  evaluation 
in  future  ship  acquisition  programs  will  be  ad- 
dressed on  a case-by-case  basis. 

Why  was  the  decision  made  not  to  require 
whole  ship  operational  test  and  evaluation  on 
the  CGN  38  and  FFG  7 classes?  Certainly  cost 
was  an  important  factor.  In  both  cases  studied, 
the  minimum  out-of-pocket  dollar  cost  to  the 
Navy  to  do  whole  ship  operational  test  and 
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evaluation  was  estimated  to  be  less  than  one 
half  of  one  percent  of  total  program  cost,  de- 
pending on  costing  assumptions.  However,  if 
even  this  modern  cost  becomes  an  unfunded  re- 
quirement the  Ship  Acquisition  Program  Man- 
ager has  another  significant  management  prob- 
lem to  solve!  Two  other  factors  probably  had 
greater  influence  on  the  decision,  the  compre- 
hensive development  test  and  evaluation  and 
operational  test  and  evaluation  plans  outlined 
in  the  CGN  38  and  FFG  7 TEMPs  (and  cur- 
rently being  executed),  and  the  fact  that  the  re- 
sult of  whole-ship  operational  test  and  evalua- 
tion, if  it  were  conducted,  probably  would  not 
have  significant  impact  on  changing  the  many 
ships  of  the  class  already  contracted  for  without 
incurring  exorbitant  cost  and  schedule  penal- 
ties. In  addition,  the  many  weeks  that  a ship  is 
' dedicated  to  whole-ship  testing,  it  is  not  avail- 
able to  the  Fleet  Commander  for  filling  forward 
deployment  commitments. 

SUMMARY 

Ship  test  and  evaluation  policy  recognizes 
that  ships  are  different  from  most  other  weapon 
system  acquisition — the  difference  being  that 
the  long  design,  engineering,  and  construction 
period  for  a major  warship  will  normally  pre- 
clude testing  the  lead-ship  in  a class  prior  to  the 
decision  to  proceed  with  follow-ships.  The  pur- 
pose of  ship  test  and  evaluation,  at  least  for  con- 
ventional, displacement-hull  warships,  it  to  re- 
duce risks  rapidly,  and  minimize  the  need  for 
modification  to  follow-ships  before  deploy- 
ment. The  policy  presumes  a commitment  to 
the  ship  acquisition  program,  and  that  Mile- 
stone IIIA  will  occur,  once  sufficient  perfor- 
mance has  been  demonstrated  to  reduce  techni- 
cal and  operational  risks  to  a manageable  level. 
For  ship  combat  systems,  the  policy  is  detailed, 
considered,  and  flexible. 

The  thrust  of  the  policy  for  new  ship  types, 
those  incorporating  major  technical  advance- 
ments not  earlier  proved  in  hull  and  nonnuclear 
propulsion  design,  appears  to  focus  test  and 
evaluation  on  technical  advancement.  This  pol- 
icy could  be  interpreted  to  mean  that  an  ad- 
vanced technology,  high-speed  prototype  plat- 
form (not  a warship)  capable  of  zipping  about 
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the  ocean’s  surface,  but  lacking  a warship’s 
combat  system,  is  adequate  to  address  all  the 
operational risksbeiote  Milestone  III.  It  is  not. 
The  final  sentence  in  the  directive  on  the  subject 
of  ship  propulsion  system  innovations  clearly 
implies  that  the  po'icy  is  concerned  with  ac- 
quisition of  warships,  not  just  high  performance 
platforms: 

“Adequate  test  and  evaluation  on  such 

prototype  [ship]  will  be  completed 

prior  to  the  first  major  production  dec- 
ision on  follow  ships.”' 

In  addition  to  directing  a program  of  early, 
at-sea  follow-on  operational  test  and  evaluation 
in  the  lead  ship  for  specified  systems  and  equip- 
ments on  all  new  ship  classes,  the  policy  goes 
one  step  further  in  regard  to  conventional, 
displacement-hull  warships.  The  whole-ship 
operational  test  and  evaluation  concept  “...if  re- 
quired, full  ship  operational  evaluation  to  the 
degree  feasible...”  seems  incongruous  with  the 
policy  premise  of  rapidly  reducing  risks  so  as  to 
minimize  the  need  to  modify  follow-ships  in  a 
class,  at  least  until  we  learn  how  to  shrink  the 
many  years  needed  to  design,  engineer,  and 
construct  the  lead  warship. 

Almost  certainly,  additional  knowledge 
about  the  new  warship’s  capability  to  serve  the 
Navy  mission  in  the  long-term  and  short-term 
could  be  generated  by  carrying  out  the  whole- 
ship  operational  test  and  evaluation.  Perhaps 
the  few  remaining  operational  unknown-un- 
knowns might  be  gleaned,  but  these  should 
have  been  ferreted  out  through  vigorous  opera- 
tional test  and  evaluation  on  the  combat  and 
propulsion  systems  in  the  normal  lead-ship  op- 
erational test  and  evaluation  shortly  after  deliv- 
ery. What  probably  cannot  be  obtained  by 
COMOPTEVFOR  without  the  rigor  of  a 
whole-ship  operational  evaluation  are  solid  an- 
swers to  long-term  evaluation  questions,  or 
warship  performance  in  a vast  range  of 
fieet/threat  scenarios  and  ocean  environments. 
Given  the  10  or  more  years  from  initiation  of  a 
major  warship  acquisition  program  until  results 
of  a whole-ship  operational  test  and  evaluation 
are  available  to  the  Chief  of  Naval  Operations, 
the  value  of  the  policy  as  a useful  management 
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tool  for  rapid  risk  reduction  during  acquisition 
seems  questionable.  The  resources  expended  in 
such  a whole-ship  operational  evaluation  might 
be  better  spent  in  careful  engineering  and  test 
and  evaluation  before  Milestone  IIIA. 

PRACTICE 

When  the  new  ship  test  and  evaluation  pol- 
icy was  published  in  early  1973,  the  Navy  had 
already  structured  two  major  ship  acquisition 
initiatives  to  comply  fully  with  this  policy.*  The 
patrol  missile  hydrofoil  program  was  to  use  a 
prototype  ship  (PHM  1)  for  operational  evalua- 
tion before  the  first  major  production  decision 
at  Milestone  III.  The  FFG  7 program  had  both 
a propulsion  system  and  a combat  system  land- 
based  test  site  under  construction,  as  well  as  ex- 
tensive planning  underway  for  component 
equipment  and  subsystem  OPEVALS  on  surro- 
gate ships.  Timing  of  the  operational  test  and 
evaluation  for  over  nine  major  systems  and 
their  integration  was  orchestrated  to  ensure 
that  comprehensive  evaluations  would  be  avail- 
able to  the  Chief  of  Naval  Operations  at  the 
critical  Milestone  IIIA  decision  for  the  fellow- 
ships. 

Even  though  the  production  PHM  and 
FFG  7 class  warships  are  still  years  away  from 
deployed  fleet  service,  it  is  fair  to  say  that  oper- 
ational test  and  evaluation  has  shown  a return 


•PHM  class  and  FFG  7 class  ship  acquisition  programs 
were  structured  under  the  new  test  and  evaluation  policy. 
Five  ship  class  acquisition  programs  (CV'N  68,  SSN  688, 
DD  963,  LHA  1,  and  CGN  38),  all  past  Milestone  IIIA, 
were  designated  by  the  Chief  of  Naval  Operations  to 
comply  with  the  new  test  and  evaluation  policy  contained 
in  DOD  Directive  5000.3  for  ships,  with  minimum  cost 
and  schedule  impact.  Compliance  consisted  of  “back- 
fitting”  the  new  policy  to  include  follow-on  test  and 
evaluation  where  a need  existed,  e.g.,  operational  risk, 
complete  approval  for  service  use  process,  demonstrated 
operational  effectiveness  and  operational  suitability  of 
integrated  combat  or  propulsion  systems.  Five  very  differ- 
ent follow-on  test  and  evaluation  programs  resulted,  rang- 
ing in  scope  from  a few  COMOPTEVFOR  officers  wit- 
nessing the  lead  ship's  Acceptance  Trials  and  shakedown 
training  evolutions,  to  minioperational  evaluations  of  new 
systems  and  extensive  combat  system  integration  tests. 


on  the  investment.  Final  judgment  on  the  value 
of  initial  operational  test  and  evaluation  con- 
ducted as  part  of  the  PHM  and  FFG  7 pro- 
grams cannot  be  rendered  until  the  ships  have 
been  through  several  years  of  fleet  service. 
Major  warship  acquisition  programs  initiated 
after  PHM  and  FFG  7 programs  are  structured 
using  a similar  demonstrated-performance/ 
milestone  strategy. 

Whereas  the  FFG  7 program  faced  a single 
critical  test  and  evaluation  milestone,  the  new 
AEGIS  ship  acquisition  programs  are  sched- 
uled for  a sequence  of  four  significant 
engineering/test  events  (Operational  Test  IIIA, 
B,  C,  and  D),  starting  with  basic  weapon  testing 
at  sea  and  culminating  in  operational  test  and 
evaluation  of  the  integrated  combat  system  at 
the  land  based  test  site  before  the  full  ship 
production  release  milestone.  Follow-on  opera- 
tional test  and  evaluation  on  the  class  lead-ship 
should  complete  the  operational  test  and  evalu- 
ation process. 


Prior  to  the  “try  before  buy”  policy,  a Ship 
Acquisition  Program  Manager’s  performance 
and  motivation  had  been  measured  by  the  ship 
he  delivered,  after  he  delivered  it.  Typical 
management  performance  measures  would  be 
unit  cost  for  each  ship  constructed;  whether 
scheduled  contract  delivery  dates  were  met; 
and,  how  many  design  and  production  defi- 
ciency items  were  included  in  the  report  by  the 
President,  Board  of  Inspection  and  Survey,  af- 
ter a brief  Acceptance  Trial  at  the  time  of  ship 
delivery  (years  after  the  design  decisions).  An 
operationally-oriented,  independent  evaluation 
of  the  ship’s  potential  effectiveness,  based  on 
demonstrated  performance  to  date,  was  not 
available  to  senior  Navy  decisionmakers  at  in- 
termediate points  in  the  acquisition  program. 
Operational  risks  identified  in  the  ship  design 
and  engineering  process  were  addressed 
through  a few  formal  boards  and  official  corre- 
spondence by  the  Chief  of  Naval  Operations, 
the  Naval  Material  Command,  and  the  fleet 
and  type  commanders  when  they  were  not 
overly  committed  with  their  primary  mission  of 
fleet  operation,  readiness  training,  and  mainte- 
nance. 
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Today,  the  operational  effectiveness,  oper- 
ational suitability,  need  for  change,  and  prelim- 
inary tactics  for  new  ship  classes  are  the  direct 
concern  of  COMOPTEVFOR  throughout  the 
ship  design,  engineering,  and  fleet  introduction 
phases  of  the  acquisition  process.  More  impor- 
tantly, the  COMOPTEVFOR  independent  as- 
sessment of  operational  performance  is  avail- 
able to  the  Chief  of  Naval  Operations  when  pro- 
gram milestones  dictate.  The  actual  planning, 
conducting,  and  reporting  operational  test  and 
evaluation  is  the  function  of  an  operational  test 
director  and  his  team  of  officers  and  senior 
petty  officers  who  address  the  adequacy  of 
demonstrated  warship  performance.  The  ship 
operational  test  director  is  responsible  for  draft- 
ing the  operational  test  and  evaluation  plan  for 
integration  with  development  test  and  evalua- 
tion and  production  acceptance  test  and  evalua- 
tion in  the  TEMP,  the  Chief  of  Naval 
Operations-approved  “contract”  negotiated  be- 
tween COMOPTEVFOR  and  Ship  Acquisition 
Manager.  Operational  test  and  evaluation  is  a 
planned  element  in  the  structure  of  ship  acqui- 
sition programs. 

For  the  record,  FFG  7 class  passed  Mile- 
stone IIIA  in  1975  without  the  need  for  a for- 
mal meeting  of  the  Defense  Systems  Acquisi- 
tion Review  Council  because  the  Navy  had 
done  its  homework  in  operational  test  and  eval- 
uation. Production  of  the  NATO  Patrol  Missile 
Hydrofoil  class  was  authorized  in  1977.  When 
PHM  1 entered  the  fleet  in  the  same  year,  the 


President,  Board  of  Inspection  and  Survey  re- 
ported to  the  Chief  of  Naval  Operations  that 
PEGASUS,  (PHM  I),  was  one  of  the  best,  if  not 
the  best  trial  the  Board  had  completed. 

“...THE  BOARD  RECOGNIZES  THAT 
THE  SHIPS  HAD  BEEN  IN  SERVICE  FOR 
SOME  TIME  BEFORE  PRESENTATION, 
HOWEVER  THE  EXCELLENT  ENGI- 
NEERING OF  HER  PROPULSION  SYS- 
TEM DEMONSTRATED  SUPERB  RELIA- 
BILITY THROUGH  HER  TRIAL  WITH 
NO  MAJOR  OR  SIGNIFICANT  BREAK- 
DOWNS OR  FAILURES.  THE  COMBAT 
SYSTEM  ALSO  PERFORMED  AS  DE- 
SIGNED AND  ADVERTISED.  THE  SHIP, 
AS  DESIGNED  AND  SUPPORTED  BY 
EITHER  A MOTHER  SHIP  OR  AD- 
VANCED BASE  WITH  ITS  LOW  PROPUL- 
SION SIGNATURE,  HIGH  SPEED  AND 
MANEUVERABILITY,  ARMED  WITH  A 
HARPOON  MISSILE  SYSTEM,  HAS  A 
FORMIDABLE  CAPABILITY  FOR  SUR- 
VEILLANCE AND  MONITORING  NAR- 
ROW SEA  PASSAGES  AND  ANTI-SHIP 
ACTION  WHERE  A LARGER  UNIT’S  CA- 
PABILITIES MIGHT  BE  INAPPROPRI- 
ATE OR  WASTED...’” 

It  is  the  application  of  operational  test  and 
evaluation  and  the  involvement  of 
COMOPTEVFOR  in  ship  acquisition  pro- 
grams 6e/oreand  a/?er Milestone  III  that  is  new 
in  the  Navy  response  to  DOD  Directive  5(XX).3. 


REFERENCES 


1.  Dept  of  Defense,  “Test  and  Evaluation,”  Directive,  Number  5000.3, 19  Jan  73,  p 5. 

2.  Dept  of  Navy,  COMOPTEVFOR  to  CNO  (serial  912),  Ltr,  subject:  “Second  Partial  Report  on  CNO  Project 

179-2-OT-III,  Partial  Operational  Test  and  Evaluation  of  the  EFG-7  Propulsion  System,”  3 Sep  76. 

3.  , “Approval  of  Systems  and  Equipments  for  Service  Use,”  OPNAVINST  4720.9D:  Para  12,  “Special 

Considerations,”  23  Aug  74,  p 11. 


4 , “Land  Based  Test  Site  (LBTS)  Selection  Policy.”  NAVMATINST  3960.8, 1 Sep  76. 

5.  Cdr  Karl  M.  Duff,  USN,  LCDR  Henry  Schmidt,  Jr..  USN,  and  IXTDR  Michael  R.  Terry,  USN,  “The  NATO 
Patrol  Missile  Hydrofoil  (PHM)  Ship  and  Weapons  Systems  Technical  Evaluation  Program,”  Technical  Paper, 
Preprint  No.  76-848,  presented  at  Third  Advanced  Marine  Vehicles  Conference  sponsored  by  American  Insti- 
tute of  Aeronautics  and  Astronautics,  Inc.,  and  Society  of  Naval  Architects  and  Marine  Engineers,  Arlington, 
VA,  20-22  Sep  76. 


Vol  I,  No.  S 


43 


6.  Dept  of  Navy,  COMOPTEVFOR  to  CNO  (OP-03),  Ltr,  subject;  “Operational  Appraisal  Plan  for  FFG-7,"  30 

Oct  75. 

7.  . , President,  Board  of  Inspection  and  Survey  (PRESINSORV)  to  Chief  of  Naval  Operations,  message, 

(131748Z)  Jun77. 


1 


,^'^i 

Commander  Ian  E.  M. 

Donovan  is  assigned  to  the 
Ship  Evaluation  Division  at 
COMOPTEVFOR  as  the 
Operational  Test  Coordina- 
tor for  USS  VIRGINIA 
(CGN  38)  ship  class  and  the 
new  AEGIS  cruiser  class 
(CGN  42).  Sea  experience  of  Commander  Donovan  in- 
cludes deck,  gunnery,  engineering  and  operations  duties  in 
USS  CONE  (DD  866),  USS  SELLSTROM  (DER  255), 
USS  FORRESTAL  (CVA  59),  and  USS  STICKELL  (DD 
888).  He  also  served  in  USS  BIDDLE  (CG  34)  and  USS 
SELLSTROM  as  Executive  Officer.  For  13  months  in 
Vietnam  he  was  Force  Plans  Officer  on  the  staff  of 
COMNAVFORV/CHNAVADVGRP  MACV. 

Commander  Donovan  is  a graduate  of  the  DES- 
LANT Engineering  School,  Defense  Intelligence  School, 
and  School  of  Naval  Command  and  Staff,  Newport,  RI. 
While  on  the  staff  at  the  US  Naval  War  College,  Cdr  Do- 
novan was  Director,  International  Law  Study,  a Research 
Program  Officer,  and  Director,  Senior  Officer  Executive 
Management  Course.  At  the  NROTC  Unit,  University  of 
Louisville,  KY,  he  instructed  midshipmen  in  naval  engi- 
neering and  management. 

Commander  Donovan  received  his  commission 
through  the  Regular  NROTC  program  when  he  gradu- 
ated from  The  Ohio  State  University  with  a BS  in  Business 
Administration  (1955).  He  received  his  MS  degree.  Inter- 
national Affairs,  from  The  George  Washington  Univer- 
sity. 
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Lieutenant  Commander 
Thomas  A.  Fitzgibbons  is 
Engineer  Officer,  USS  TRI- 
POLI (LPH  2).  He  recently 
completed  a 2-year  tour  at 
COMOPTEVFOR  where  he 
was  the  Operational  Test  Di- 
rector for  DD  963  class  de- 
stroyers. Prior  assignments  include  service  as  Chief  Engi- 
neer with  the  commissioning  crew,  USS  FANNING  (FF 
1076)  and  a tour  of  duty  in  Vietnam  as  a Riverine  Warfare 
Advisor.  In  his  first  tour  of  duty  he  served  as  Main  Pro- 
pulsion Assistant  and  Damage  Control  Assistant  in  USS 
SOUTHERLAND  (DD  743). 

Lieutenant  Commander  Fitzgibbons  graduated  from 
the  US  Naval  Academy,  class  of  1966.  He  later  completed 
Destroyer  School  (Department  Head  Course).  LCDR 
Fitzgibbons  received  a masters  degree  in  Weapon  Systems 
Acquisition  Management  from  the  Naval  Postgraduate 
School. 
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DEFENSE  PROCUREMENT: 


THE  BRITISH  CONNECTION 

by 

Captain  T.  H.  Sherman,  United  States  Navy 


For  many  years  the  United  States  has  encouraged  standardization  of  weapons  systems 
among  our  Allies,  principally  within  the  North  Atlantic  Treaty  Organization  (NATO). 
However,  an  examination  of  the  weapons  system  inventory  of  the  United  States  reveals  tew 
foreign  systems.  Conversely,  US  equipment  frequently  appears  in  foreign  inventories.  The 
situation  has  fostered  the  opinion,  held  by  many  abroad,  that  the  United  States  supports 
standardization  only  so  long  as  it  applies  to  purchases  of  US  defense  equipment.  While  US 
Allies  acknowledge  that  the  US  Services  have  been  limited  in  purchasing  equipments  abroad  by 
the  “Buy  American  Act,”  (47  Stat.  1530;  41  U.S.  Code)  the  perception  is  that  the  US  Armed 
Services  have  a reluctance  to  purchase  any  systems  “not  invented  here.” 


A MEMORANDUM 
OF  UNDERSTANDING 

Standardization  within  NATO  has  been  a of  this  two-way  street  with  Britain  implies  a 

goal  for  at  least  three  recent  Secretaries  of  commitment  by  the  United  States  to  allow 

Defense.  Secretary  Schlesinger  made  many  pro-  foreign  suppliers  to  bid  in  the  United  States 

nouncements  of  this  goal  and  began  negotia-  defense  market  on  equal  footing  with  United 

tions  with  the  United  Kingdom.  In  September  States  industry.  Procurement  from  other  Euro- 

1975,  Secretary  Schlesinger  concluded  a Mem-  pean  Allies  with  whom  the  US  does  not  have  a 

orandum  of  Understanding  (MOU)  with  his  similar  MOU  is  handled  on  a case-by-case  ba- 

British  counterpart  regarding  cooperation  in  sis.  In  these  cases  when  NATO  standardization 

reciprocal  defense  procurement.  The  primary  will  be  enhanced,  the  US  may  waive  the  “Buy 

purpose  of  the  agreement  is  to  promote  greater  American”  price  differentials. 

United  States — United  Kingdom  (UK)  cooper- 
ation in  Research  and  Development  production  The  MOU  is  broad  in  scope  and  describes 

and  procurement  to  enhance  NATO  rational-  the  principles  to  be  followed  by  the  United 

ization  and  standardization,  and  to  achieve  the  States  and  Great  Britain  in  reciprocal  procure- 

greatest  NATO  capability  at  the  lowest  possible  ment.  Included  is  a provision  for  the  develop- 

cost.  With  certain  exceptions,  the  MOU  per-  ment  of  test  and  evaluation  procedures  by  the 

mits  the  industry  of  each  country  to  compete  in  Office  of  the  Secretary  of  Defense  (OSD)  in 

the  defense  market  of  the  other  on  an  equal  cooperation  with  the  United  Kingdom.  The 

basis.  An  ultimate  result  of  the  agreement  will  Director  of  Defense  Research  and  Engineering 

be  a long  term  equitable  balance  in  defense  is  responsible  for  research,  development,  test 

trade.  As  a government  department  head,  with  and  evaluation  related  to  bilateral  arrange- 

authority  delegated  by  Congress,  the  Secretary  ments  for  joint  military  development.  Subse- 

of  Defense  has  exempted  defense  items  from  the  quent  implementing  instructions  to  the  Services 

restrictions  of  the  Buy  American  Legislation  address  the  issue  of  test  and  evaluation  (T&E) 

for  the  purposes  of  this  agreement.  The  opening  in  providing  for  the  consideration  of  UK  items 


Vol  I,  No.  5 


45 


which  have  been  tested  and  accepted  by  the  UK 
for  Service  use.  An  appraisal  of  the  sufficiency 
of  UK  testing  for  US  purposes  is  to  be  on  a case- 
by-case  basis,  and,  where  retesting  is  necessary. 
United  States  test  and  evaluation  standards, 
policies,  and  procedures  will  apply.  The  negoti- 
ations concerning  the  test  and  evaluation  as- 
pects of  the  MOU  are  currently  under  discus- 
sion and  questions  concerning  these  negotia- 
tions have  not  been  fully  resolved. 

The  question  of  acceptability  for  Service 
use  is  a most  important  aspect  of  the  US/UK 
memorandum  and  the  implementing  instruc- 
tions. For  example.  United  States’  Services 
have  definite  test  and  evaluation  disciplines  to 
which  new  hardware  is  subjected.  Included  are 
development  and  operational  testing,  both  of 
which  ultimately  lead  to  a judgment  on  item 
acceptability  for  Service  use.  One  asks  the  ques- 
tion, "How  closely  do  UK  test  and  evaluation 
disciplines  parallel  those  of  the  US?  Is  it  possi- 
ble for  the  United  States  to  relate  the  various 
phases  of  UK  testing  to  US  standards?  If  so, 
should  an  equipment  or  system  which  has  ma- 
tured in  the  UK,  and  which  has  been  in  the  field 
with  a UK  Service,  be  subjected  to  the  full 
gamut  of  US  testing  or  can  the  process  be 
abbreviated?”  Recently  there  have  been  exam- 
ples of  mature  UK  systems  having  been  dis- 
qualified by  US  procurement  activities  based  on 
inadequate  testing  (in  US  eyes)  when  in  fact 
extensive  testing  and  Service  use  had  been  ac- 
complished in  the  United  Kingdom.  Most 
probably  this  disqualification  occurred  as  the 
result  of  ignorance  of  British  testing  and  its 
applicability  to  the  US  procurement  process. 
How  to  reconcile  these  inconsistencies  and  im- 
plement the  MOU  in  a timely  manner  is  a 
current  task  of  the  Deputy  Director,  Defense 
Research  and  Engineering  (Test  and 
Evaluation).  To  determine  the  range  of  this 
effort  an  examination  of  the  DOD  test  philoso- 
phy and  definition  as  stated  in  DOD  Instruc- 
tion 5000.3  is  in  order. 

A new  system  under  development  under- 
goes many  tests,  from  subsystem  to  system, 
either  in  a laboratory  environment  or  in  the 
field.  This  Development  Test  and  Evaluation 
(DT&E)  is  an  integral  part  of  advanced  and 


engineering  development.  The  developer  uti- 
lizes this  testing  to  validate  design,  sound  out 
the  envelope  of  capability  and  demonstrate  reli- 
ability growth.  At  some  point  the  developer  and 
the  contractor  reach  a level  of  confidence  that 
the  system  can  be  operated  and  maintained  by 
US  soldiers,  sailors,  or  airmen  (without  outside 
help)  and  is  ready  for  the  field.  At  that  time,  the 
Materiel  Command  certifies  that  the  system  is 
ready  for  a military  operational  determination 
of  system  acceptability  for  Service  use. 

With  development  certification  achieved, 
the  system  is  passed  to  the  cognizant  Service’s 
independent  test  agency  for  Operational  Test 
and  Evaluation  (OT&E).  These  tests  are 
planned  and  conducted  by  the  agency,  although 
the  actual  evaluations  may  be  performed  by 
representative  field  units  under  cognizant 
agency  control.  The  tests  are  made  without 
contractor  support  as  the  field  unit  evaluates 
the  system  for  the  Service  Chief.  Test  condi- 
tions are  selected  to  be  as  realistic  as  possible, 
including  stress  environment  such  as  would  be 
experienced  in  wartime.  Thus,  the  developing 
agency  and  the  contractor  are  taken  out  of  the 
loop  and  the  system  undergoes  a graduation 
exercise  conducted  by  a stern  judge.  Note  that 
while  this  judge  is  in  uniform,  he  is  not  serving 
as  the  ultimate  user,  the  troops  in  the  field.  This 
independence  from  both  developer  and  user  in 
testing  and  reporting  is  most  important.  Inde- 
pendent test  reports  concerning  the  operational 
effectiveness  and  suitability  of  the  system  go 
directly  to  the  respective  Service  Chief  of  Staff 
and  do  not  imply  the  urgency  to  field  the 
hardware  that  the  user  might  feel  nor  do  the 
reports  reflect  the  advocacy  of  the  developer.  In 
short,  the  independent  tester  is  the  referee  be- 
tween the  developer  and  user. 

The  test  and  evaluation  procedures  fol- 
lowed by  the  Services  involve  extensive  re- 
sources of  money,  time  and  personnel.  As 
experience  is  gained,  more  efficient  means  of 
accomplishing  test  and  evaluation  goals  are 
sought  and  implemented.  For  example,  while 
the  independent  tester  has  been,  in  the  past, 
relatively  nonin volved  in  developmental  test- 
ing, valuable  resources  can  be  saved  if  the 
independent  tester  participates  in  and  keeps 
well  informed  of  the  developer’s  testing.  In  this 
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manner,  redundant  testing  is  avoided  during 
Operational  Test  and  Evaluation  and  the  devel- 
oper can  better  prepare  the  system  for  the 
operational  tests  that  will  be  imposed. 

Finally,  if  testing  has  been  successful  and 
other  requirements  of  need,  resources  and  risk 
reduction  have  been  met,  the  system  may  be 
released  for  production.  Even  then,  testing  may 
continue  after  production  to  proof  design 
changes  and  backfits  required  to  correct  previ- 
ous test  deficiencies  or,  when  appropriate,  to 
contribute  to  modernization  of  fielded 
equipments. 

Discussions  with  the  Ministry  of  Defence 
were  begun  in  Fall  1976  to  determine  the  appli- 
cability of  United  Kingdom  testing  to  United 
States  procurement  policies.  These  discussions 
were  informal  and  were  intended  to  review  the 
problem  and  to  outline  a course  of  action  for 
the  future.  First,  it  was  agreed  that  negotiations 
would  be  concerned  only  with  complete  defense 
systems  as  opposed  to  a full  shopping  list  of 
subsystems  and  components.  While  United 
Kingdom  defense  production  quantities  are 
much  less  than  those  of  the  United  States,  it 
should  be  possible  to  relate  the  US  concept  of  a 
“major  system”  to  that  of  the  United  Kingdom. 
Second,  it  was  apparent  that  each  side  should 
become  familiar  with  the  test  and  evaluation 
policies  and  procedures  of  the  other  in  some 
detail.  Relevant  Office  of  the  Secretary  of 
Defense/Ministry  of  Defence  (OSD/MOD) 
documentation  was  exchanged  for  further 
study.  Reciprocal  visits  to  test  and  evaluation 
agencies  were  proposed  to  facilitate  agreement 
on  terminology,  procedures,  equivalence,  and 
perhaps  a Test  and  Evaluation  annex  to  the 
MOU. 

DIFFERENCES 

On  the  US  side  it  was  apparent  from  the 
documentation  that  two  basic  differences  in  test 
and  evaluation  philosophy  set  the  United  States 
and  the  United  Kingdom  apart,  aside  from  the 
expected  differences  in  nomenclature. 

1.  The  UK  does  not  have  a coun- 
terpart to  the  US  independent 

Service  test  agencies  (Commander, 


Operational  Test  and  Evaluation 
Force  (COMOPTEVFOR),  Op- 
erational Test  and  Evaluation 
Agency  (OTEA),  and  Air  Force 
Test  and  Evaluation  Center 
(AFTEO).  All  testing  is  the 
responsibility  of  the  developer. 

The  United  States  policy  stresses 
the  independence  of  the  opera- 
tional tester  from  the  developer 
and  user.  The  British  Army  more 
nearly  approaches  the  US  concept 
of  the  independent  tester  by  utili- 
zation of  special  boards  that  ob- 
serve and  report  on  trials. 

2.  Most  operational  testing  in  the 
UK  Services  (by  US  definition)  is 
performed  after  a decision  is  made 
to  go  into  production. 

To  place  these  differences  in  proper  per- 
spective, note  that  in  Britain  the  procurement 
process  is  extremely  complex  and  demanding  of 
a Program  Manager.  Procurement  milestones 
are  more  numerous  and  often  more  itringenl 
than  those  imposed  by  the  US  Department  of 
Defense  on  the  three  US  Services.  Further, 
British  defense  resources  are  severely  limited  in 
both  people  and  money.  For  these  reasons  the 
British  feel  that  the  UK  cannot  afford  a sepa- 
rate test  and  evaluation  hierarchy  and  count  on 
the  integrity  of  the  present  system  to  ensure 
adequate  test  and  evaluation.  Since  the  UK 
production  runs  are  orders  of  magnitude 
smaller  than  those  of  the  US,  fewer  test  articles 
are  available.  Consequently  United  Kingdom 
test  programs  are  less  ambitious  than  those  of 
the  United  States.  Operational  testing  must 
await  a commitment  to  production  to  obtain 
test  articles.  The  British  Operational  testing  is 
comparable  to  US  Follow-on  Test  and  Evalua- 
tion (FOTE).  Feedback  from  FOTE  occurs  and 
since  the  United  Kingdom  inventory  is  small, 
fewer  items  require  backfitting  when  discrepan- 
cies are  discovered  by  testing.  The  British  find 
this  method  of  defense  system  test  and  evalua- 
tion to  be  cost  effective  and  within  resources. 

The  first  formal  meeting  in  the  negotia- 
tions consisted  of  a visit  by  a representative  of 
DD(T&E)  to  Britain  for  the  purpose  of  seeing 


first-hand  the  test  and  evaluation  activities  of 
the  United  Kingdom  Ministry  of  Defence.  The 
purpose  was  to  determine  the  similarities  and 
differences  between  the  United  States  and  the 
United  Kingdom  systems.  On  completion  of 
the  visit,  a meeting  was  held  to  explore  ways  to 
reconcile  existing  differences  that  might  hinder 
reciprocal  purchasings.  The  conclusions 
reached  were; 

• In  the  United  Kingdom,  test  and  evalu- 
ation is  an  integral  part  of  project  de- 
velopment. 

• The  United  Kingdom  project  direc- 
tors/managers have  full  responsibility 
for  all  aspects  of  test  and  evaluation 
during  development  and  for  recom- 
mending system  readiness  for  Service 
use.  As  a consequence,  United  King- 
dom test  and  evaluation  (UK  T&E), 
prior  to  production  release,  would  be 
considered  in  the  context  of  DODD 
5000.3  as  limited  to  Development  Test 
and  Evaluation. 

• Operational  testing  is  usually  con- 
ducted after  the  production  decision  is 
made  by  the  UK  Service  involved.  Par- 
ticular groups  in  the  three  Services  are 
not  involved  in  Operational  Test  and 
Evaluation  on  all  projects.  Participa- 
tion in  such  testing  is  generally  on  a 
case-by-case  basis. 

• Rigorous  adherence  to  common  test 
and  evaluation  procedures  does  not 
exist  among  the  UK  Services;  in  fact, 
procedures  may  vary  among  projects 
within  a single  Service. 

Formal  presentation  of  these  conclusions 
was  made  in  a joint  paper,  agreed  upon  between 
the  British  Ministry  of  Defence  and  the  United 
States  DD(T&E).  The  paper  has  been  submit- 
ted to  higher  authority  in  both  countries.  The 
intent  is  that  this  joint  paper  be  the  basis  for  a 
Test  and  Evaluation  Annex  to  the  MOU. 

The  identification  of  differences  only  sur- 
faces potential  problem  areas.  It  is  necessary 


that  these  differences  be  evaluated  in  terms  of 
the  limitations,  if  any,  that  the  differences  may 
impose  on  reciprocal  procurement.  Solutions 
must  be  found  to  accommodate  the  intent  of  the 
MOU.  In  ( rtain  situations  the  US  Services 
might  be  required  to  adjust  United  States  pro- 
cedures to  accommodate  a United  Kingdom 
system  being  considered.  In  other  situations  an 
adjustment  in  United  Kingdom  testing  should 
be  made.  It  is  evident  that  precise  rules  and 
pr(X.edures  cannot  be  laid  down  for  all  systems. 
Each  prospective  foreign  procurement  should 
be  evaluated  on  a case-by-case  basis.  These 
preliminary  meetings  aired  the  problem  and 
indicated  possible  routes  by  which  US/UK  test 
and  evaluation  activities  can  be  made  compati- 
ble For  example,  some  progress  was  made  in 
regard  to  test  and  evaluation  reporting.  Owing 
to  resource  limitations,  the  United  Kingdom 
feels  that  a reorganization  within  the  Ministry 
of  Defence  to  include  an  independent  Test  and 
Evaluation  Agency  as  in  the  United  States  is 
not  practicable.  Further,  the  British  belief  is 
that  individual  UK  Service  test  and  evaluation 
should  continue  in  its  present  form.  However,  it 
was  agreed  that  the  United  Kingdom  should 
establish  a single  point-of-contact,  a project 
manager,  for  test  and  evaluation  matters  rele- 
vant to  reciprocal  procurement.  The  project 
manager  will  be  responsible  for  all  liaison  be- 
tween the  United  States  and  the  United  King- 
dom Services  for  reporting  test  and  evaluation 
status  and  results.  In  the  interest  of  standard- 
ization and  for  the  purpose  of  facilitating  this 
exchange,  the  United  Kingdom  will  provide 
test  and  evaluation  data  in  the  format  of  the 
existing  United  States  Test  and  Evaluation 
Master  Plan  (the  TEMP).  In  this  manner,  the 
United  States  counterpart  should  be  able  to 
correctly  relate  United  Kingdom  testing  to 
United  States  requirements.  At  present  it  is 
extremely  difficult  for  an  American  to  correlate 
United  States  testing  requirements  with  those 
of  the  United  Kingdom.  The  TEMP  format 
should  provide  an  “audit  trail”  for  US  person- 
nel use  in  following  UK  test  and  evaluation  and 
to  evaluate  format  adequacy  for  US  purposes. 
An  essential  part  of  reciprocal  procurement  of 
newly  emergent  systems  appears  to  be  early 
involvement  by  both  seller  and  purchaser.  For 
example,  the  test  and  evaluation  plan  for  a 
system  under  development  in  Britain  may  fully 
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satisfy  UK  test  and  evaluation  requirements 
but  fall  short  of  certain  special  requirements  of 
the  United  States.  The  United  States  may  re- 
quire further  testing  in  unusual  environments 
of  weather,  sea  state,  electronic  countermea- 
sures, etc.  It  may  be  possible  for  the  United 
States  purchaser  to  specify  additional  require- 
ments early  enough  in  the  United  Kingdom 
development  cycle  so  that  these  requirements 
may  be  satisfied  in  conjunction  with  United 
Kingdom  testing.  At  least,  the  United  States 
purchaser  will  be  well  aware  of  those  areas 
where  additional  United  States  testing  will  be 
required  to  qualify  the  system  for  United  States 
Service  use. 

It  would  seem  that  the  more  common  case 
regarding  cooperation  in  reciprocal  defense 
procurement  and  the  one  easiest  to  deal  with, 
would  be  a United  Kingdom  system  in 
production  which  is  already  Service  approved. 
With  the  test  and  evaluation  data  in  TEMP 
format,  the  United  States  purchaser  can  readily 
determine  those  areas,  if  any,  that  will  require 
additional  testing  by  the  United  States. 

Given  that  a United  Kingdom  system 
meets,  or  can  meet  by  additional  testing.  United 
States  requirements  for  procurement,  a final 
hurdlii  must  be  achieved  on  this  side  of  the 
Atlantic.  The  United  States  must  obtain  suffi- 
cient test  articles  for  the  US  designated  inde- 
pendent service  test  agency  to  conduct  some 
degree  of  Follow-on  Operational  Testing  to 
establish  operational  effectiveness  and  opera- 
tional suitability,  including  interoperability 
with  other  US  systems.  Most  probably  this 
Follow-on  Test  and  Evaluation  will  reveal  areas 
of  deficiency.  Hence  a feed-back  loop  for  cor- 
rections by  the  United  Kingdom  supplier  is 
essential.  The  extent  of  this  Follow-on  Test  and 
Evaluation  will  vary  dependent  upon  how  early 
in  the  United  Kingdom  development  cycle  the 
United  States  service  test  agency  becomes  in- 
volved. Advice  given  by  the  United  States  ser- 
vice test  agency  to  the  United  Kingdom  devel- 
oper should  make  a significant  reduction  in  the 
need  for  redundant  testing. 

As  a means  of  familiarizing  United  King- 
dom counterparts  with  the  unique  US  concept 


of  independent  operational  testing,  a team  of 
Ministry  of  Defence  officers  and  civilians  was 
invited  to  the  United  States  in  June  1977.  The 
team  members  visited  and  were  briefed  by  each 
of  the  US  Service  independent  test  agencies.  In 
addition  the  team  members  witnessed  opera- 
tional testing  in  the  field  performed  by  each 
agency.  Although  limited  in  scope  by  current 
test  and  evaluation  schedules,  the  team  mem- 
bers saw  testing  of  the  Army’s  Light  Air  Cush- 
ion Vehicle  (LACV)  at  Fort  Story,  Virginia;  the 
Navy’s  Patrol  Frigate  propulsion  test  site  at 
Philadelphia  and  the  Land  Based  Combat  Sys- 
tem Test  Site  at  Islip,  New  York,  followed  by 
Air  Force  testing  of  the  F-16  at  Edwards  Air 
Force  Base,  California. 

Based  on  this  experience,  the  United 
Kingdom  team  members  felt  that  much  of  the 
United  Kingdom  follow-on  operational  testing 
parallels  that  of  the  United  States  and  that  in 
large  degree  United  Kingdom  testing  should 
satisfy  United  States  requirements.  To  this  end, 
the  British  extended  an  invitation  for  a compa- 
rable US  team  to  visit  the  United  Kingdom  in 
Autumn  1977  for  visits  to  United  Kingdom  test 
activities  and  meeting  with  the  three  Service 
counterparts.  These  visits  serve  the  purpose  of 
gaining  appreciation  of  each  other’s  scope  of 
operation  and  testing  philosophy  while  at  the 
same  time  establishing  individual  points-of- 
contact  for  future  liaison  and  coordination  on 
test  ana  evaluation  matters  (as  these  matters 
relate  to  reciprocal  procurement). 

The  negotiations  should  culminate  in  the 
Test  and  Evaluation  Annex  to  the  MOU  that 
will  contain  jointly  agreed  upon  policies  and 
procedures.  The  policies  and  procedures  will 
most  likely  appear  as  broad  guidelines  for  each 
service  to  apply  on  a case-by-case  basis  rather 
than  as  specific  procedures.  In  this  manner,  ad- 
justments can  be  made  to  accommodate  the 
wide  variance  of  test  and  evaluation  in  the 
United  Kingdom  Services  and  systems. 

SUMMARY 

Although  the  idea  of  military  standardiza- 
tion among  US  Allies  has  been  with  us  for  a 
long  time,  the  United  States  has  only  recently 
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taken  the  lead  to  make  it  a reality.  The  MOU 
with  Britain  should  usher  more  United  King- 
dom systems  into  the  United  States  defense 
market  on  equal  footing  with  US  suppliers  and 
be  a forerunner  for  agreements  with  other  na- 
tions. The  MOU  with  Britain  will  not  open  a 
floodgate  of  candidates  for  the  US  market; 
there  are  simply  not  that  many  systems  in 
question.  The  MOU  will  ensure,  however,  that 
the  “not  invented  here”  philosophy  will  not 
prevent  an  opportunity  for  United  States  forces 
to  obtain  superior  United  Kingdom  defense 
products  which  may  be  available.  Further,  the 
MOU  should  eventually  enable  an  equitable 
balance  in  defense  purchasing  between  the  two 
nations. 
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OPTIMIZING  SPACE  VEHICLE 
TEST  PROGRAMS 

by 

Dr.  George  J.  Gleghorn 
Assistant  General  Manager  for  Engineering 
Space  Systems  Division 
TRW  Defense  and  Space  Systems  Group 


Spacecraft  have  an  important  and  significant  role  in  CS  national  defense.  Spacecraft  now 
provide  surveillance,  navigation,  weather  prediction,  and  communications  capabilities.  The 
testing  and  evaluation  of  spacecraft  involves  difficulties  not  usually  encountered  in  test 
programs  of  other  defense  systems  such  as  missiles,  aircraft,  armored  vehicles,  naval  vessels  or 
ordnance. 

To  date  space  systems  testing  has  been  an  expensive  activity.  The  high  premium  placed  on 
“success  the  first  time”  is  reflected  in  high  cost.  Space  flight  and  operation  of  remote  sensing 
and  communications  equipment  in  space  are  new  undertakings.  In  a typical  program,  the 
spacecraft  test  and  evaluation  activities  require  from  7 months  to  more  than  1 year  and  absorb 
about  10  percent  of  the  total  program  funding  for  development  and  acquisition. 


CONSTRAINTS  ON  THE 
SPACE  VEHICLE 

Inaccessibility 

The  outstanding  characteristic  of  past  and 
present  day  spacecraft  is  the  current  lack  of  a 
capability  to  perform  maintenance  while  the 
spacecraft  is  in  orbit.  This  factor  drives  the 
design  in  the  direction  of  never  fail  and  away 
from  free  access,  preventive  maintenance,  and 
the  ease  of  repair  and  replacement  concepts 
found  in  typical  aircraft,  computer  and  weapon 
systems.  Once  a .spacecraft  has  lifted  off  the 
launch  pad,  it  is  no  longer  accessible  for  failure 
diagnosis,  maintenance  or  repair.  Even  in 
manned  spacecraft  there  are  severe  limitations 
on  the  capability  for  performing  the  visual 
inspections  and  manual  repair  that  is  taken  for 
granted  with  most  other  defense  systems. 

Not  only  is  repair  or  maintenance  impossi- 
ble, but  also  any  direct  observation,  perfor- 
mance evaluation  or  change  in  operating  mode 


not  designed  into  the  spacecraft  is  also  impossi- 
ble. The  only  access  to  the  spacecraft  after 
liftoff  is  through  telemetry  and  the  radio  fre- 
quency command  link.  The  integrity  of  radio 
communications  with  the  spacecraft  is  of  vital 
importance.  The  use  of  this  communications 
link  in  the  ground  test  program  follows  from 
the  basic  objective  of  the  test  program,  which  is 
to  demonstrate  that  the  spacecraft  is  capable  of 
operating  as  desired  in  space. 

Delivery  Cost 

Each  spacecraft  requires  a delivery  system 
that  places  the  spacecraft  into  the  proper  orbit. 
The  cost  of  the  nonrecoverable  boost  vehicle 
and  the  launch  support  costs  are  large,  that  is, 
never  less  than  from  $8  to  $10  million  and 
ranging  to  upwards  of  $50  million.  Hence, 
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delivery  of  extra  spacecraft  to  replace  a space- 
craft that  has  failed  in  orbit  generally  represents 
an  excessive  economic  burden.  A lower  cost 
approach  is  to  provide  sufficient  testing  and 
“burn-in”  prior  to  launch  to  eliminate  all  weak 
comp<.ments  which  are  prone  to  early  failure. 
Thus  the  high  delivery  cost  provides  strong 
motivation  for  an  effective  and  exhaustive 
ground  test  and  evaluation  program.  Despite 
the  fact  that  use  of  the  space  shuttle  will  reduce 
the  delivery  cost,  especially  for  low  earth  orbit 
missions,  it  still  remains  costly  enough  to  sus- 
tain the  requirement  for  an  adequate  test 
program. 

Space  Environment 

The  unique  characteristics  of  the  space 
environment  and  the  intense  vibration,  acous- 
tic, and  gravitational  stresses  imposed  on  the 
spacecraft  during  boost  have  a significant  im- 
pact on  the  spacecraft  configuration  and  the 
verification  test  program.  Equipment  must 
work  in  a permanent,  hard  vacuum  that  has  a 
strong  effect  on  design.  One  cannot  depend  on 
corrective  airflow  for  temperature  control. 
Rather  the  designs  depend  upon  analysis  of 
heat  conduction  and  radiation  paths  and  care- 
ful control  of  heat  loss  to  space  by  means  of 
insulation  and  surface  treatment  of  radiating 
panels. 

Hard  vacuum  also  has  its  effect  on  lubrica- 
tion design.  Lubricants  must  be  sealed  in,  or 
special  vacuum  lubricants  must  be  used.  Hard 
vacuum  has  its  effect  on  radio  frequency  design. 
Often  transmitters  must  be  able  to  radiate  at  all 
pressures  from  atmosphere  to  full  vacuum.  The 
potential  of  corona  discharge  must  be  consid- 
ered in  this  full  regime. 

Furthermore,  equipment  must  work  in  a 
free  fall,  zero-G  environment.  This  has  benefits 
as  well  as  disadvantages:  For  example,  a struc- 
ture for  booms  and  appendages  can  be  made 
very  light  as  it  need  not  support  weight.  Con- 
versely dependency  cannot  ^ placed  on  gravity 
as  in  ground  operations.  Example:  fuel  must  be 
forced  through  the  engines,  and  mechanisms 
must  be  positively  activated  by  springs  or 
motors. 


Other  environmental  factors  that  severely 
impact  the  design  of  a spacecraft  include  the 
charged  particle  fluxes  encountered  in  the  radi- 
ation belts  and  the  special  environment 
produced  by  remote  nuclear  detonations  in  the 
atmosphere  or  in  space.  Rapid  recovery  from 
transient  weapons  effects  and  immunity  from 
permanent  damage  are  required  characteristics 
that  can  be  achieved  only  through  special  de- 
sign techniques.  The  design  characteristics 
must  be  validated  through  ground  testing  that 
attempts  to  simulate  the  weapons  effects. 

The  natural  radiation  environment  is  still 
being  explored  by  a variety  of  scientific  satel- 
lites and  space  probes.  The  effects  caused  by  the 
charged  particle  streams  during  geomagnetic 
substorms  have  been  particularly  troublesome 
to  certain  satellites  in  geostationary  orbits.  Spe- 
cial design  precautions  are  used  to  prevent 
surface  arc  discharges  because  of  the  environ- 
ment encountered  during  substorms. 

UNIQUE  CHARACTERISTICS 

The  severe  constraints  placed  on  satellites 
because  of  remote  positioning,  the  space  envi- 
ronment and  the  difficulty  of  getting  satellites 
into  space  lead  to  a number  of  unique  charac- 
teristics and  practices  which  are  not  found  in 
combination  in  any  other  defense  system. 

Remote  Monitoring  and  Control 

The  inaccessibility  of  the  spacecraft  re- 
quires that  all  payload  generated  data  and  all 
diagnostic  measurements  be  transmitted  to 
earth  for  interpretation  and  evaluation  over  a 
suitable  radio  frequency  downlink.  In  addition 
an  uplink  is  provided  to  transmit  commands 
from  earth  to  the  spacecraft.  Thus  every  space- 
craft contains  a Tracking,  Telemetry  and  Com- 
mand (TT&C)  subsystem  that  also  can  be  used 
to  great  advantage  for  ground  checkout  of  the 
spacecraft.  In  typical  military  spacecraft  the 
TT&C  subsystem  will  accommodate  more  than 
1000  different  encoded  commands  and  provide 
several  hundred  diagnostic  performance  mea- 
surements and  status  signals.  Although  this  is 
adequate  to  control  the  spacecraft  from  the 
ground  under  normal  unfailed  conditions  and 
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to  diagnose  failures  for  which  work  arounds 
have  been  designed  into  the  spacecraft,  it  gener- 
ally is  not  sufficient  to  permit  acceptance  test- 
ing of  the  assembled  spacecraft  prior  to  launch. 
Additional  hardline  test  points  and  external 
stimulators  must  be  provided  for  ground 
testing. 

Component  Redundancy 

Active  or  standby  component  redundancy 
has  been  one  of  the  most  potent  methods  for 
achieving  a high  probability  of  mission  success 
after  delivery  of  a satellite  into  orbit.  Electronic 
components  and  many  mechanical  components 
usually  have  a similar  redundant  component 
aboard  that  takes  over  automatically  or  can  be 
engaged  by  ground  command  if  the  prime  com- 
ponent fails.  In  many  instances  there  is  more 
than  one  redundant  component  or  there  is  an 
alternate  method  of  executing  the  full  mission 
or  a reduced  mission.  This  is  how  the  require- 
ment for  “graceful  degradation”  has  been  im- 
plemented. The  ground  test  program  must  ver- 
ify that  failure  of  a component  can  be  detected, 
that  failures  are  not  propagated  to  unfailed 
elements,  and  that  the  telemetry  and  command 
subsystem  is  capable  of  providing  continuing 
operation  by  means  of  the  proper  redundant 
component. 

A difficulty  in  considering  such  redundant 
designs  and  long  design  lives  is  that  one  cannot 
directly  verify  the  design  by  test.  It  is  difficult  to 
determine,  objectively,  how  to  speed  up  the 
testing.  Life  testing  can  be  done  on  certain 
components  of  the  system,  but  economics  and 
program  priorities  dictate  that  an  entire  satel- 
lite cannot  be  tested  for  years  prior  to  the  time 
the  decision  to  go  into  production  is  made. 
Thus,  another  set  of  extrapolations  must  be 
depended  upon. 

The  result  is  that  reliability  is  designed 
into  the  satellite  by  calculation,  by  incorporat- 
ing redundancy,  by  depending  upon  piece  part 
reliability  and  by  exhaustive  testing  of  each 
part.  It  <s  common  practice  to  do  qualification 
testing  cf  each  manufacturing  lot  of  parts.  In 
addition,  extensive  bum-in  and  screening  of 


parts  for  parameter  drift  may  be  required.  Ex- 
pensive as  this  treatment  of  the  problem  is,  it  is 
cheaper  than  is  part  failure  after  installation. 

The  strategy  of  using  redundancy  to  pro- 
vide mission  reliability  for  long  periods  of  time, 
such  as  7 to  10  years,  has  not  been  used  to  this 
extent  on  other  defense  systems  where  the  unit 
cost  is  low  or  where  maintenance  and  repair  are 
possible. 

Physical  Characteristics 

The  boost  vehicle  imposes  severe  weight 
and  volume  constraints  on  the  space  vehicle  it 
places  into  orbit.  Our  largest  military  boost 
vehicle,  the  Titan  III,  can  place  about  3200 
pounds  into  geostationary  orbit  at  a cost  in 
excess  of  $40  million.  During  boost  the  space- 
craft must  fit  under  the  aerodynamic  shroud 
and  hence  must  fit  into  a cylindrical  volume  10 
feet  in  diameter.  The  cheaper  Atlas/Centaur 
boost  vehicle  can  boost  about  4200  pounds  into 
a transfer  orbit  at  a cost  of  more  than  $20 
million.  With  an  additional  propulsion  motor 
that  fires  at  the  apogee  of  the  transfer  orbit,  the 
Atlas/Centaur  provides  the  capability  to  put  a 
2000  pound  spacecraft  into  geostationary  orbit. 

Designing  for  these  boosters  requires  that 
large  spacecraft  antennas,  solar  arrays,  and 
other  appendages  be  folded  to  fit  into  the 
booster  shroud  and  be  deployed  sometime  after 
the  shroud  is  jettisoned  and  the  high-G  portion 
of  the  boost  phase  is  completed.  The  basic  point 
is  that  the  test  and  evaluation  program  must  be 
designed  to  verify  the  validity  and  reliability  of 
the  deployment  process  under  close  simulation 
of  the  zero-gravity  conditions. 

STRATEGIES  FOR  SPACECRAFT 
TESTING  AND  EVALUATION 

In  view  of  the  operating  constraints  and 
the  unique  combination  of  design  features, 
what  then  makes  sense  for  test  and  evaluation 
of  a satellite  program?  In  practice,  the  program 
for  each  satellite  development  is  tailored  to  the 
state  of  development  of  the  hardware  involved. 
Such  a program  generally  consists  of  four  parts: 
development,  qualification,  acceptance,  and 
prelaunch  validation  testing. 
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! A uniform  set  of  definitions  and  typical 

■ requirements  for  each  part  of  the  test  program 

i are  contained  in  M1L-STD-1540A,  “Test  Re- 

I quirements  for  Space  Vehicles.”  This  standard 

I has  been  prepared  to  serve  as  a guideline  docu- 

ment for  procuring  agencies  and  spacecraft 
contractors. 

Development  testing  depends  heavily  on 
the  type  of  hardware,  parts,  and  components  to 
be  used  and  the  variety  and  novelty  of  op- 
erational requirements.  A development  test 
program  may  include  life  tests,  design  margin 
tests,  failure  mode  tests,  and  all  sorts  of  design 
verification  tests.  Extensive  testing  of  engineer- 
ing and  developmental  models  is  done  to  verify 
concepts  and  performance  within  constraints. 
Of  particular  importance  is  the  verification  of 
structural  integrity,  mechanisms,  and  thermal 
system  design.  Wherever  possible  development 
tests  are  performed  at  the  lowest  level  of  assem- 
bly where  results  are  meaningful  and  applicable 
to  the  performance  of  the  assembly  as  part  of 
the  spacecraft.  Thus  antennas,  for  example,  can 
be  tested  without  being  mounted  to  the  space- 
craft, solar  array  designs  can  be  validated  by 
measuring  the  electrical  output  of  the  repre- 
sentative modular  sections  and  by  checking 
structural  and  thermal  performance  on  a 
mockup  that  uses  simulated  solar  cells.  Stabil- 
ity of  the  attitude  control  subsystem  can  be 
analyzed  by  using  a single  axis  rate  table  and 
simulators  instead  of  a complete  spacecraft  and 
elaborate  3-axis  test  equipment.  Preliminary 
thermal  system  design  verification  is  obtained 
from  an  analytical  model  that  relies  on  mea- 
sured values  of  emittance  and  absorption  coeffi- 
cients and  estimates  of  heat  loads.  This  ap- 
proach, that  includes  component  testing,  math- 
ematical modeling  based  on  key  measurements, 
extensive  breadboard  testing,  and  subsystem 
testing,  provides  confidence  that  subsequent 
qualification  testing  of  a complete  spacecraft 
will  not  require  extensive  retest  and  redesign. 
The  approach  saves  schedule  time  because  de- 
velopment and  test  of  different  subsystems  can 
proceed  in  parallel. 

Life  testing  is  done  usually  only  for  those 
devices  which  are  expected  to  have  a wearout 
mechanism:  bearings,  lubrication  systems,  tape 


recorders,  vacuum  tubes  (travelling  wave  tube 
amplifiers  are  an  example),  solar  cells,  and  the 
like.  Often  the  life  testing  is  accelerated  when  a 
life  dependence  of  some  parameter  is  known — 
but  for  some  devices  acceleration  is  not  feasible 
and  dependence  must  be  placed  on  extrapolat- 
ing trends  from  rea'.iime  life  testing.  Such  is  the 
case  for  travelling  wave  tubes  and  for  bearings 
exposed  to  vacuum  conditions. 

It  is  a common  practice  to  devote  one 
vehicle  to  qualification  testing.  That  is  where, 
in  the  test  program,  the  functional  performance 
and  the  margin  of  safety  against  environmental 
exposures  are  demonstrated.  Typical  exposures 
are  acoustic  and  vibration  extremes  (to  simulate 
the  rocket  powered  flight  regime)  and  space 
simulation  (to  simulate  the  vacuum  and  ther- 
mal interface  conditions  encountered  in  orbit). 

Space  simulation  testing  takes  many  days 
to  complete  because  the  thermal  time  constants 
are  long.  This  means  that  this  type  of  testing 
constitutes  the  most  severe  test,  that  is,  continu- 
ous operation  without  access  for  vehicle  repair, 
adjustment  or  maintenance.  Recognition  of  the 
usefulness  of  such  a test  in  elimination  of  flawed 
equipment  has  led  some  programs  to  require 
long  term  operation  (say  30  days)  with  no  out- 
of-tolerance  performance  as  a final  acceptance 
test. 


Acceptance  testing  for  each  satellite  is 
done  to  confirm  that  workmanship  is  adequate 
and  that  the  equipment  functions,  in  detail,  as 
expected.  Acceptance  tests  include  end-to-end 
functional  tests  that  exercise  every  subsystem; 
an  acoustic  or  vibration  test;  a thermal  vacuum 
test;  and,  a high  pressure  test  of  any  spacecraft 
pneumatic  or  hydraulic  lines  or  pressurization 
equipment.  After  acceptance  testing,  the  space- 
craft is  ready  for  shipment  to  the  launch  site. 

Prelaunch  validation  tests  are  conducted 
at  the  launch  site  to  demonstrate 

• that  the  vehicle  was  not  damaged  in 
shipping, 

• that  the  spacecraft  was  properly  mated 
to  the  launch  vehicle,  and 


54 


Defense  Systems  Management  Review 


• that  the  spacecraft  can  be  controlled 
from  the  ground,  which  generally 
means  from  the  Air  Force  Satellite 
Control  Facility  (AFSCF)  and  the 
launch  base  facilities. 

The  above  test  program,  in  conjunction 
with  strict  configuration  control,  is  part  of  the 
logical  sequence  of  test  and  evaluation  that 
demonstrates  each  satellite  will  perform  its  mis- 
sion in  space. 

Qualification  and  acceptance  testing  of 
space  vehicles  requires  some  elaborate  and 
unique  test  facilities  and  instrumentation.  A 
typical  thermal  vacuum  test  setup  is  shown  in 
Figure  1.  The  FLTSATCOM  spacecraft  is 
shown  (without  solar  arrays)  in  the  30  foot 
spherical  space  simulation  chamber  as  is  a por- 
tion of  the  ground  support  equipment  needed  to 
operate  the  spacecraft.  Thermal  instrumen- 
tation, data  processing  equipment  and  chamber 
controls  are  not  shown.  A photograph  of  a 22  x 
40  foot  bottom  loading  space  environmental 
chamber  is  shown  in  Figure  2.  An  acoustic  test 
facility  that  can  provide  a sound  pressure  of  120 
to  154  decibels  is  shown  in  Figure  3. 
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Figure  2.  Environmental  Test  Chamber 


Figure  1.  Thermal  Vacuum  Test  Setup 


Figure  3.  Acoustic  Test  Configuration 
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Figure  4 shows  a deployment  test  facility  for 
large  structural  elements  such  as  solar  arrays. 


Figure  4.  Deployment  Test  Setup 


Functional  performance  testing  of  a space- 
craft makes  use  of  the  onboard  telemetry  and 
command  subsystem  and  hardline  connections 
to  critical  test  points.  Because  of  severe  weight 
limitations,  built-in  test  equipment  and  moni- 
tors that  do  not  serve  a necessary  function  after 
lift-off  are  minimal.  Most  electrical  compo- 
nents have  a test  connector  that  allows  check- 
out using  special  component  test  equipment 
and  a nonflight  test  harness  during  spacecraft 
assembly.  The  spacecraft  has  a set  of  test  con- 
nectors that  are  accessible  during  onstand 
checkout.  These  connectors  bring  out  critical 
test  points  (typically  300  for  the  FLTSATCOM 
spacecraft)  and  provide  for  connection  of  the 
ground  power  supply  that  charges  the  space- 
craft batteries  and  powers  the  spacecraft  bus.  In 
addition,  two  long  wires  that  are  interlaced 
with  the  spacecraft  harness  are  used  to  detect 
any  abnormal  electrical  noise  owing  to  electro- 
magnetic interference  or  switching  transients. 

During  thermal/vacuum  testing,  nonflight 
test  thermocouples,  thermistors  and  radiome- 
ters are  located  throughout  the  spacecraft  and 
chamber  to  verify  the  thermal  design  under  a 
variety  of  operating  conditions. 

Optical  stimulators  for  checkout  of  the 
attitude  control  subsystem  are  mounted  to  sim- 
ulate the  earth  limb,  sun  and  star  signals  that 
serve  as  attitude  control  references. 


Radio  frequency  measurements  are  made 
through  use  of  a variety  of  receiving  antennas, 
coaxial  cables  and  waveguide  connections  to 
the  spacecraft. 

Comprehensive  functional  spacecraft 
checkout  requires  that  every  operating  mode 
and  every  component,  switch  and  redundancy 
path  be  exercised  to  verify  integrity.  In  a typical 
spacecraft  there  are  more  than  1000  possible 
ground  commands,  a similar  number  of  teleme- 
try measurements  and  several  hundred  temper- 
atures to  be  monitored  during  ground  testing. 
Thus  the  use  of  computers  for  data  processing 
and  command  sequencing  is  essential  to  achieve 
repeatable  test  conditions  and  economy  of 
schedule. 

The  set  of  electrical  ground  support  equip- 
ments for  functional  checkout  of  a typical  com- 
munication satellite  such  as  FLTSAT  consists 
of  the  following: 

• Electrical  Power  Test  Set 

The  Electrical  Power  Test  Set  provides 
electrical  power,  battery  charge  con- 
trol, simulation  of  solar  array  output 
and  display  of  critical  spacecraft  volt- 
ages and  currents. 

• Attitude/Velocity  Control  Test 
Set 

The  Attitude/Velocity  Control  Test 
Set  provides  artificial  stimulation  of 
the  spacecraft’s  attitude  control  sen- 
sors. This  test  set  also  monitors  re- 
sponse of  reaction  wheels,  thrusters 
and  solar  array  drive  motors  through 
telemetry  or  hardline  connections  for 
each  operating  mode  of  the  attitude 
and  velocity  control  subsystem. 

• Tt&C  Test  Equipment 

The  Tracking,  Telemetry  and  Com- 
mand Radio  Frequency  Console, 
through  coax  lines,  transmits  com- 
mand uplink  signals  and  receives  te- 
lemetry downlink  signals  to  the 
spacecraft. 
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• TT&C  Test  Set 

An  additional  test  set  contains  the  digi- 
tal data  processing  encrypting  and 
recording  equipment  for  encoding  and 
decoding  telemetry  and  command 
data. 

• Automatic  Communications  Test 
Equipment 

The  Automatic  Communications  Test 
Equipment  performs  all  communica- 
tions payload  tests.  Commercial  RF 
test  equipment  is  used.  This  equipment 
is  controlled  by  a minicomputer  with  a 
capability  of  storing  test  sequences  for 
about  20(X)  separate  tests,  using  over 
500  commands.  Tabulation  and  plot- 
ting of  test  results  in  near  real  time 
allows  for  assessing  progress  as  the  test 
proceeds  and  for  flexibility  of  testing. 

• Automatic  Data  Processing 
Equipment 

The  Automatic  Data  Processing 
Equipment  consists  of  a small  real  time 
digital  computer  with  64000  byte 
memory  capacity  and  is  used  to  proc- 
ess and  display  measurements,  provide 
out-of-limits  warnings,  generate  com- 
mands and  command  sequences,  store 
calibration  data  and  provide  personnel 
access  for  control  of  the  test. 

• Thermal  Test  Equipment 

The  Thermal  Test  Equipment  controls 
special  heat  sources  used  during  ther- 
mal vacuum  testing,  multiplexes  test 
thermocouple  or  thermistor  outputs 
and  contains  a separate  computer  that 
proce.s,ses  and  displays  thermal 
measurements. 

Techniques  for  reducing  the  cost  and  dura- 
tion of  the  test  program  have  been  developed 
over  the  years  as  automatic  data  processing 
equipment  has  become  available  and  the  tech- 
nology of  spacecraft  design  has  matured.  Com- 
puter hardware,  peripheral  equipment  and 


computer  software  have  been  designed  so  that 
payload  tests  and  certain  spacecraft  subsystem 
tests  can  be  carried  out  in  parallel.  Commonly 
used  computers  and  software  languages  mini- 
mize startup  costs.  Use  of  centralized  comput- 
ers for  the  testing  of  different  spacecraft  in  the 
same  assembly  and  test  area  have  been  consid- 
ered. The  fact  that  spacecraft  can  be  expected  to 
become  more  complicated  while  at  the  same 
time  requiring  longer  checkout  and  burn-in 
testing  will  require  further  automation  of  test 
procedures. 

MANAGING  T&E  PROGRAMS 

Specialized  or  custom  design,  low  produc- 
tion rates,  high  reliability  and  other  unique  as- 
pects of  space  vehicle  programs  impose  require- 
ments for  significant  management  trade  off 
among  cost,  schedule  and  technical  risk  in  the 
planning  and  implementation  of  test  and  evalu- 
ation programs.  Although  the  test  and  evalua- 
tion experience  gained  over  the  past  two  de- 
cades now  is  reflected  in  a number  of  directives, 
standards,  and  specifications,  there  is  a flexibili- 
ty within  this  framework  to  custom  design  each 
test  program,  to  make  appropriate  cost,  sched- 
ule and  risk  trades  to  achieve  maximum  confi- 
dence that  the  design  and  workmanship  will 
meet  mission  requirements. 

Several  .significant  considerations  that  can 
affect  cost,  schedule,  or  performance  risk  are 
discussed  as  examples  of  tradeoffs  in  the  test 
program. 

• To  be  effective  in  influencing  design  so 
that  the  vehicle  can  be  adequately 
tested,  and  to  reduce  schedule,  one 
should  commit  to  and  start  implemen- 
tation of  a test  philosophy  and  selec- 
tion of  test  equipment  very  early  in  the 
program.  However,  the  earlier  the  de- 
sign of  test  equipment  is  begun,  the 
more  vulnerable  it  is  to  design  changes 
imposed  by  changes  in  design  of  the 
vehicle.  Cost  and  schedules  are  ad- 
versely impacted  when  these  changes 
occur. 

• The  very  low  number  of  production 
articles  in  each  program  requires  a 
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different  way  of  looking  at  how  general 
purpose  facilities,  equipment,  tooling, 
etc.,  will  be  supplied.  Since  each  pro- 
gram is  specifically  tailored  to  its 
unique  mission,  the  amount  of  special 
purpose  equipment  and  specialized 
computer  software  tends  to  be  dispro- 
portionately high  with  respect  to  capi- 
tal investment  and  as  a percentage  of 
total  program  costs.  The  challenge  is  to 
increase  general  purpose  capability 
without  jeopardizing  the  ability  to  ac- 
complish specialized  testing. 


• Since  test  and  evaluation  starts  at  the 
piece  part  level  and  continues  through 
component,  subsystem  and  system 
level  tests,  cost  and  schedule  can  be 
reduced  by  the  selective  performance 
of  environmental,  burn-in,  electrical 
compatibility  and  other  types  of  tests 
only  at  one  or  two  of  the  assembly 
stages,  rather  than  at  each  of  the  four 
stages.  Such  cost  and  schedule  reduc- 
tions must  be  accomplished  without 
adding  significant  risk. 

• The  constraints  imposed  by  the  space 
operating  environment  and  the  pres- 
sures to  keep  spacecraft  weight  down 
may  result  in  a spacecraft  that  is  rela- 
tively fragile,  easily  damaged,  and  dif- 
ficult to  maintain  because  of  packing 
density.  To  reduce  the  possibility  of 
damage,  safety  awareness  programs 
and  training  need  to  be  continuously 
worked,  protective  coverings  and  de- 
vices have  to  be  used  for  sensitive  com- 
ponents, and  highly  disciplined  and 
procedural  test  operations  arc  re- 
quired. These  requirements  add  sched- 
ule and  cost  constraints  that  must  be 
traded  off  against  potentially  serious 
damage  to  the  test  article. 

Although  there  are  many  considerations 
which  are  amenable  to  tradeoff,  there  is  a signif- 
icant body  of  experience  that  verifies  standard- 
ization is  a cost  effective  goal.  Thus  every  test 
should  be  thoroughly  planned  in  terms  of  objec- 
tives, data  requirements,  criteria  for  success. 


etc.  Step-by-step  procedures  that  call  for  qual- 
ity assurance  sign-off  at  all  critical  steps  as  the 
steps  are  performed  are  issued  to  test  personnel 
well  in  advance  of  each  test.  Records  are  kept  of 
all  measurements  and  all  anomalies  or  “out-of- 
spec”  conditions.  The  data  are  evaluated  by 
knowledgeable  personnel  for  tell-tale  trends  af- 
ter each  major  test  or  environmental  exposure 
to  determine  any  incipient  failures  that  require 
repair  or  replacement  of  equipment  prior  to 
launch. 

The  overall  management  of  test  programs, 
especially  for  the  larger  and  more  complex 
systems,  is  enhanced  by  utilization  of  earned 
value  and  other  advanced  program  manage- 
ment techniques  that  have  been  developed  and 
proved  over  this  past  decade.  The  large  invest- 
ment represented  by  the  test  items,  the  many 
engineering  internal  and  external  interfaces  in- 
volved, and  above  all  the  ever  present  require- 
ment for  long  term,  fault-free  reliability  de- 
mand that  detailed  cost  effective  management 
be  exercised  at  all  times.  Nothing  can  be  left  to 
chance. 
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TRW  Defense  and  Space 
Systems  Group.  Dr.  Gleg- 
horn  joined  TRW,  Inc.,  in 
1954.  He  engaged  in  System 
Engineering  and  Technical  Direction  on  the  ballistic  mis- 
sile and  space  h<x»ier  programs — Atlas,  Thor,  Titan. 
From  1958  to  the  present  he  has  worked  in  the  manage- 
ment of  many  ;.alellile  projects.  Early  in  1960  he  was 
Program  Manager  for  the  Orbiting  Geophysical  Observa- 
tory. Dr.  Gleghorn  has  been  in  a supervisory  management 
position  for  such  satellites  as  Pioneer,  Vela,  Intel.sat  HI, 
DSCS  II,  and  FleelSatCom,  For  a 2-year  pericxl  Dr. 
Gleghorn  was  manager  of  the  TRW  System  Integration 
and  Test  Operations. 

Pnor  to  his  affiliation  with  TRW,  Inc.,  Dr.  Gleghorn 
worked  for  several  years  at  Hughes  Aircraft  and  the  Jet 
Propulsion  Laboratory.  He  served  2 years  during  the 
Korean  War  in  the  US  Navy  (destroyers). 

Dr.  George  J.  Gleghorn  received  his  B.S.  from  the 
University  of  Colorado.  He  performed  his  graduate  work 
at  Cal  Tech. 
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MANAGEMENT  OF  MAJOR  DOD 
TEST  FACILITIES 


by 

John  W.  McCord,  Director,  Test  Resources 
Office  of  the  Undersecretary  of  Defense  Research  & Engineering 


INTRODUCTION 

The  Department  of  Defense  Major  Range 
and  Test  Facility  Base  (MRTFB)  comprises  26 
test  facilities  under  the  management  of  the 
Army,  Navy  and  Air  Force.  See  Table  I.  Within 
the  Office  of  the  Secretary  of  Defense  (OSD), 
responsibilities  for  the  MRTFB  reside  with  the 
Deputy  Undersecretary  of  Defense  Research 
and  Engineering  (Test  and  Evaluation) 
(DUSDR&E(T&E)).  Office  of  the  Undersecre- 
tary of  Defense  Reseaich  & Engineering.  Sup- 
port is  provided  by  the  Director  for  Test  Re- 
sources. Key  responsibilities  intlude 

• providing  overall  policy  directii.n  and 
guidance, 

• insuring  adequacy  to  meet  present  and 
future  requirements,  and 

• achieving  optimum  utilization  of  range 
assets. 

Within  each  Service,  test  facility  manage- 
ment and  administrative  activities  take  place  at 
the  headquarters  (Army  Staff,  Chief  of  Naval 
Operations,  Air  Staff)  and  systems  command 
(Army  TECOM,  Naval  Air  Systems  Com- 
mand, and  Air  Force  Systems  Command)  lev- 
els, as  well  as  at  the  facility  command  level.  The 
activities  include  defining  specific  missions, 
policies  and  plans  to  guide  facility  development 
and  structuring  the  facilities  to  support  specific 
kinds  of  test  activities  and  internal  test  pro- 
grams, joint  DOD  test  programs  and,  other 
facility  user  requirements  within  the  capability 


of  the  facilities.  The  Service  must  plan  and 
budget  for  all  facility  costs  and,  with  a few 
exceptions,  fund  all  indirect  costs,  i.e.,  those 
costs  not  directly  attributable  to  a user.  The 
facility  commander  is  required  to  develop  and 
maintain  a master  plan  for  the  operation  and 
development  of  the  facility  and  act  as  a general 
“plant  manager.” 

The  status  of  range  assets  such  as  real  pro- 
perty, data  acquisition  and  processing  equip- 
ment, and  mobile  platforms  is  continually  bal- 
anced against  test  support  requirements.  Be- 
cause of  budget  constraints  range  improvement 
and  modernization  programs  are  driven  by  the 
needs  rather  than  the  desires  of  concerned  acti- 
vities. The  entire  spectrum  of  test  resources  of 
;'li  Services  is  viewed  with  the  intention  of  satis- 
fying total  Department  of  Defense  (DOD)  re- 
quirenicnts  with  minimum  duplication.  The 
Deputy  Undersecretary  of  Defense  Research 
and  Engineering  (Test  and  Evaluation) 
(DUSDR&E(T&E))  is  the  focal  point  within- 
DOD  to  resolve  major  problems  concerning  en- 
croachment and  the  availability  of  adequate 
land,  sea,  airspace,  and  electromagnetic  envi- 
ronments to  support  DOD  Testing. 

A series  of  studies  aimed  at  consolidating 
test  functions  to  effect  cost  savings  was  initiated 
during  FY  1977.  These  actions  are  supported 
by  recent  Congressional  guidance  and  General 
Accounting  Office  (GAO)  recommendations. 
Also,  in  past  years  savings  were  realized  by 
closing  unneeded  facilities,  deactivating  surplus 
instrumented  ships  and  aircraft,  and  retiring 
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ARMY 

Cold  Regions  Test  Center 

Fort  Greely,  Alaska 

White  Sands  Missile  Range 

New  Mexico 

Tropic  Test  Center 

Fort  Clayton,  Canal  Zone 

Kwajalein  Missile  Range 

Marshall  Island,  Pacific 

Yuma  Proving  Ground 

Yuma,  Arizona 

Electronic  Proving  Ground 

Fort  Huachuca,  Arizona 

Jefferson  Proving  Ground 

Madison,  Indiana 

Dugx'ay  Proving  Ground 

Salt  Lake  City,  Utah 

Aberdeen  Proving  Ground 

Aberdeen,  Maryland 

NAVY 

Pacific  Missile  Test  Center 

Point  Mugu,  California 

Naval  Air  Engineering  Center 

Lakehurst,  New  Jersey 

Atlantic  Undersea  T&E  ,„enter 

Andros  Island , Bahamas 

National  Parachute  Test  Range 

El  Centro,  California 

Naval  Air  Test  Center 

Patuxent  River,  Maryland 

Naval  Weapons  Center 

China  Lake,  California 

Naval  Air  Propulsion  Test  Center 
Trenton,  New  Jersey 

Atlantic  Fleet  Weapons  Training 

Facility 

Roosevelt  Roads,  Puerto  Rico 

AIR  FORCE 

Space  & Missile  Test  Center 

Vandenberg  AFB,  California 

Flight  Test  Center 

Edwards  AFB,  California 

Eastern  Test  Range 

Patrick  AFB,  Florida 

Armament  Development  & Test  Center 

Eglin  AFB,  Florida 

Satellite  Control  Facility 

Sunnyvale,  California 

Air  Defense  Weapons  Center 

Tyndall  AFB,  Florida 

Tactical  Fighter  Weapons  Center 

Nellis  AFB,  Nevada 

Arnold  Engineering  Development  Center 
Tullahoma,  Tennessee 

4950th  Test  Wing 

Wright-Patterson  AFB,  Ohio 

Table  1,  Department  of  Defense  Major  Range  and  Test  Facility  Bases 
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obsolete  instrumentation.  A recent  study  lead- 
ing to  a system  of  uniform  measures  of  work- 
load, capacity,  and  capabilities  should  allow 
accurate  planning  at  DOD  test  facilities.  Such  a 
system  will  assist  management  levels  in  analyz- 
ing resource  utilization  and  provide  a valuable 
tool  to  achieve  maximum  efficiency  in  the 
MRTFB.  In  accomplishing  the  above  func- 
tions, close  liaison  is  necessary  between  key 
OSD  and  Service  elements  as  well  as  with  non- 
Defense  agencies  such  as  the  Office  of  Manage- 
ment and  Budget  (OMB),  the  National  Aero- 
nautics and  Space  Administration  (NASA),  the 
Energy  Research  and  Development  Agency 
(ERDA),  the  Federal  Aviation  Administration 
(FAA),  the  Department  of  the  Interior  (DOI), 
and  the  Department  of  Transportation  (DOT). 

POLICY  AND  REVIEW 

DOD  Directive  3200.11 

Defense  policy  for  the  use,  management 
and  operation  of  the  MRTFB  is  established  by 
DOD  Directive  3200. 1 1 . The  directive  applies 
to  the  Office  of  the  Secretary  of  Defense,  the 
Organization  of  the  Joint  Chiefs  of  Staff,  the 
Military  Departments,  Unified  and  Specified 
Commands,  and  Defense  Agencies.  The  stated 
objective  of  DOD  Directive  3200. 1 1 is: 

“...to  insure  provision  of  effective  test 
and  operational  support  by  facilitating 
joint  use  of  the  MRTFB,  by  consolida- 
ting and  standardizing  management  re- 
sponsibility at  appropriate  levels,  and 
by  setting  forth  uniform  operating 
guidelines.” 

The  directive  designates  the  26  major 
DOD  ranges  and  test  facilities  that  compose  the 
MRTFB.  These  facilities  were  selected  after  a 
comprehensive  DOD  Test  and  Evaluation  Fac- 
ility Base  Review  accomplished  in  1971  to  de- 
termine which  test  and  evaluation  ranges  and 
centers  were  essential  to  Defense  needs.  Some 
94  test  activities  were  reviewed  following  a 
screening  process  that  considered  an  initial  list 
of  several  hundred  T&E-related  activities.  The 
26  original  MRTFB  facilities  were  ultimately 
chosen,  largely  because  of  their  significant  test 


resource  assets,  multipurpose  or  multiuser  ca- 
pability and/or  unique  characteristics  or 
mission. 

Under  the  provisions  of  DOD  Directive 
3200.11,  the  composition  of  the  MRTFB  is 
periodically  reviewed  to  determine  if  changes 
should  be  considered  owing  to  current  mission 
demands  or  new  criteria.  Example:  in  1976  the 
Air  Force  Special  Weapons  Center,  Albuquer- 
que, New  Mexico,  was  deleted  from  the 
MRTFB  and  the  4950th  Test  Wing,  Wright- 
Patterson  Air  Force  Base,  Ohio,  was  added.  As 
a result  of  studies  recently  conducted  by  OSD 
and  the  Military  Departments,  it  is  envisioned 
that  additional  changes  in  MRTFB  composi- 
tion will  be  made  within  the  next  few  years. 

Additionally,  the  DOD  Directive  3200.11 
outlines  specific  responsibilities  of  the 
DUSDR&E,  Service  Secretaries,  facility  com- 
manders and  range  users. 

The  DUSDR&E  provides  policy  direction 
and  planning  guidance  and  insures  that  the 
MRTFB  is  adequate  to  meet  present  and  future 
DOD  requirements.  This  is  accomplished 
through  annual  budget  and  apportionment  re- 
views, and  continuous  oversight  of  range  opera- 
tions and  resource  needs.  The  Service  Secretar- 
ies are  designated  as  management  agents  and 
are  charged  with  defining  specific  facility  mis- 
sions, programming  and  budgeting,  and  provid- 
ing for  the  acquisition  and  replacement  of  range 
instrumentation.  The  DOD  Directive  3200. 1 1 
establishes  a Major  Range  and  Test  Facility 
Committee  (MRTFC),  chaired  by  the 
DUSDR&E(T&E)  that  includes  representa- 
tives from  each  Military  Department.  The 
Committee  provides  a vehicle  for  direct  coordi- 
nation and  rapid  resolution  of  key  issues  affec- 
ting the  MRTFB. 

The  major  burden  of  providing  test  sup- 
port to  DOD  and  other  range  users  falls  upon 
the  shoulders  of  the  facility  commander.  He  is 
responsible  for  the  actual  planning,  coordina- 
tion and  conduct  of  test  support  and  for  insur- 
ing that  alt  safety  requirements  are  met.  Range 
users  must  interface  with  the  facility  early, 
provide  adequate  planning  data  and  fund  the 


direct  costs  of  all  test  support.  The  success  of 
the  test  support  mission  is  predicated  upon  the 
timely  and  thorough  information  exchange  be- 
tween the  range  commander  and  the  user. 

The  DOD  Directive  3200. 1 1 also  defines 
the  Uniform  Funding  Policy  and  provides  for 
consideration  of  environmental  aspects  relative 
to  major  actions  affecting  the  MRTFB.  The 
directive  is  presently  undergoing  review  and 
soon  will  be  revis^  to  incorporate  recent 
organizational  and  policy  changes  within  OSD 
and  the  Military  Departments. 

Uniform  Funding  Policy 

Nineteen  of  the  26  major  DOD  ranges  and 
test  facilities,  those  possessing  the  greatest  po- 
tential for  multiservice  support,  operate  under  a 
Uniform  Funding  Policy  (UFP).  The  policy 
became  effective  in  July  1974.  Under  the  UFP, 
range  users  reimburse  the  facility  for  test  costs 
directly  related  to  their  programs.  Other  range 
expenses  are  funded  institutionally  by  the  facil- 
ity. A uniform  system  of  funding  resulted  from 
the  findings  of  the  Blue  Ribbon  Defense  Panel 
following  a series  of  studies.  Under  prior  fund- 
ing policies,  users  paid  the  full  cost  of  testing  at 
some  facilities  while  at  other  facilities  costs 
were  not  paid  in  any  amount.  At  institutionally 
funded  activities  there  was  little  incentive  for 
program  managers  to  carry  out  advanced  range 
support  planning.  In  many  instances  range  se- 
lection was  based  upon  economic  rather  than 
technical  factors.  The  various  funding  systems 
made  Service  and  OSD  management  difficult. 
The  systems  proved  inefficient  and  often  ob- 
scured the  true  cost  of  testing. 

A Joint  Logistics  Commanders  panel 
completed  reviews  of  the  UFP  in  June  1975  and 
October  1976  and  reported  that  with  a few 
exceptions  it  is  working  well.  Major  problems 
are  not  evident — minor  problems  are  being  ad- 
dressed. The  policy  proved  extremely  helpful 
during  the  justification  of  annual  range  budgets 
and  has  resulted  in  better  planning  and  commu- 
nications within  the  range  support  and  user 
communities.  The  cost  of  support  will  continue 
to  vary  in  accordance  with  type  and  location  of 
the  operation. 


Budget  Review 

The  DUSDR&E(T&E)  has  three  broad 
areas  of  budgetary  responsibility  : 

• The  Director  of  Test  and  Evaluation 
Defense  Appropriation  which  he  man- 
ages and  controls; 

• Military  Department  test  and  evalua- 
tion support  programs  for  which  he 
has  primary  OSD  cognizance;  and 

• Military  Department  major  and  se- 
lected, less-than-major  RDT&E  pro- 
grams which  he  monitors. 

The  budgetary  responsibilities  are  summarized 
below. 

Director  of  Test  and  Evaluation,  De- 
fense Appropriation 

This  separate  Defense  Appropriation 
was  established  by  Congress  in  FY 
1973  to  support  Joint  Operational  Test 
and  Evaluation  (JOT&E).  The  appro- 
priation provides  funds  for  study  ef- 
forts associated  with  the  major  DOD 
ranges  and  test  facilities.  The 
DUSDR&E(T&E)  manages  and  con- 
trols these  funds,  including  the  prepa- 
ration of  the  budget  request  for  submis- 
sion to  Congress,  testimony  before 
Congress,  and  execution  of  the 
approved  budget. 

Military  Department  T&E  Support 
Programs 

The  DUSDR&E(T&E)  reviews 
MRTFB  funding  (all  appropriations) 
in  detail  during  annual  budget,  appor- 
tionment and  POM  reviews.  In  addi- 
tion, the  Deputy  Director  has  primary 
cognizance  for  18  other  RDT&E  pro- 
gram elements  The  program  elements 
are  primarily  related  to  Service  test 
support  to  include  operation  and  main- 
tenance, overhaul  of  ships  and  aircraft, 
OT&E  capabilities,  instrumentation. 
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aerial  targets,  aircraft  survivability, 
project  SEMI,  and  other  programs. 

Military  Department  RDT&E  Programs 

The  major  responsibility  assigned  by 
the  Secretary  of  Defense  to 
DUSDR&E(T&E)  is  “monitoring 
closely  the  test  and  evaluation  planned 
and  conducted  by  the  DOD  compo- 
nents for  major  acquisition  programs 
and  for  such  other  programs  as  he  be- 
lieves necessary.”  In  FY  1978, 
DUSDR&E(T&E)  will  monitor  a total 
of  85  Army,  Navy,  Air  Force  and  De- 
fense Communications  Agency  major 
weapon  system  programs  and  15  less- 
than-major  programs. 

In  FY  1978,  budget  requests  for  test  and 
evaluation  related  activities  in  all  appropria- 
tions are  an  estimated  $2,7  billion.  The 
DUSDR&E(T&E)  is  involved  in  the  complete 
program  budget  process.  Involvement  includes 
the  formulation  and  dissemination  of  test  and 
evaluation  funding  guidance  in  coordination 
with  the  OASD  (Comptroller)  and  active  parti- 
cipation in  the  OSD/OMB  budget  and  appor- 
tionment reviews.  The  Director  for  Test  Re- 
sources performs  programming  and  budgeting 
functions  for  the  DUSDR&E(T&E)  and  main- 
tains continuous  liaison  with  elements  of  OMB, 
OSD,  and  the  Military  Departments.  Addi- 
tional budget  activities  requiring  review  and  co- 
ordination include  Congressional  Data  Sheets, 
RDT&E  Descriptive  Summaries,  Selected  Ac- 
quisition Reports,  Arms  Control  Impact  State- 
ments, GAO  Annual  Staff  Studies,  and  Con- 
gressional testimony  and  other  related  actions 
with  test  and  evaluation  impact. 

An  important  OSD  level  management 
function  is  associated  with  MRTFB  funding 
that  in  FY  1978  totals  some  $1.7  billion  from 
several  appropriations  and  includes  funds  re- 
covered from  users  for  support  provided.  Each 
Autumn  a comprehensive  review  is  conducted 
of  Service  budget  submissions  for  the  26  major 
DOD  ranges  and  test  facilities  and  the  18  other 
assigned  test  and  evaluation  support  programs. 
The  test  facility  (or  support  program),  systems 


command  and  headquarters  staffs  prepare  and 
present  the  budget.  Representatives  from 
DUSDR&E(T&E),  OASD  (Comptroller)  and 
fice  of  Management  and  Budget  perform  the 
review. 

Of  particular  interest  at  each  test  range 
and  center  are  the  improvement  and  moderni- 
zation program,  military  construction  projects, 
and  manpower  and  workload  data.  The  vali- 
dated budgetary  requirements  of  the  test  facili- 
ties and  programs  are  compared  with  overall 
fiscal  constraints,  and,  in  conjunction  with  the 
Services,  adjustments  and  offsets  are  made. 
This  program  is  monitored  during  application 
for  any  required  modifications  necessitated  by 
project  or  budget  changes. 

The  programs  are  studied  again  in  con- 
junction with  the  DOD  Apportionment  Review 
and  monitored  during  the  budget  year  through 
Service  staff  elements  and  status  reports.  Rec- 
ommendations are  made  with  respect  to  instru- 
mentation programs,  military  construction 
projects,  manning  levels  and  other  pertinent 
areas  affecting  the  MRTFB.  The  test  resource 
needs  of  the  Military  Departments  are  high- 
lighted and  priorities  are  established  for  the 
overall  test  support  community  to  meet  these 
needs  within  budgetary  limitations. 

ADEQUACY 

Range  Assets 

A principal  responsibility  of  DOD  agen- 
cies associated  with  test  resources  is  to  insure 
that  the  MRTFB  is  adequate  to  meet  current 
and  future  test  and  evaluation  requirements. 
This  responsibility  is  carried  out  by  all  levels  of 
management  to  varying  degrees  and  results  in: 

(1)  Continual  review  of  the  capabilities 
and  capacities  of  the  MRTFB  components  to 
accomplish  generic  test  and  evaluation  support 
tasks,  e.g.,  aircraft  testing,  strategic  missile  test- 
ing, electronic  systems  testing,  etc. 

(2)  Examination  of  the  types  and  amounts 
of  test  and  evaluation  workload  created  by  the 
testing  needs  of  weapon  systems  programs. 
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(3)  Comparative  analysis  of  the  MRTFB 
capability /capacity  versus  the  type/amount  of 
T&E  requirements  to  derive  both  qualitative 
and  quantitative  deficiencies  in  the  test  resource 
base. 

(4)  Development  of  plans  and  funding 
approaches  to  meet  validated  MRTFB  deficien- 
cies at  minimum  cost  to  the  institutional  test 
support  base  and  to  the  weapon  system  pro- 
grams using  this  base. 

(5)  Coordination  of  test  resource  develop- 
ment and  acquisition  projects  between  the  Mili- 
tary Departments  to  insure  that  such  projects 
achieve  the  maximum  feasible  tri-Service 
application. 

(6)  Transfer  of  test  resources  among  the 
Services  and  sponsorship  of  joint  Service  devel- 
opment and  acquisition  efforts,  where  possible. 

(7)  Interchange  of  data  associated  with  all 
Service  programs  related  to  the  development  of 
specialized  test  instrumentation  or  T&E  tech- 
niques and  methodologies  so  that  the  entire 
community  can  benefit. 

Care  is  taken  to  coordinate  all  efforts  with 
the  appropriate  agencies  outside  DOD  (e.g., 
NASA,  FAA,  DOI).  Where  required,  special- 
ized expertise  is  obtained  from  a technical  con- 
tractor. Activities  in  various  test  resource  areas 
illustrate  the  application  of  this  approach  to 
meeting  deficiencies  and  insuring  test  resource 
adequacy. 

TSPI  Systems 

The  principal  requirement  of  many  test 
exercises  conducted  within  the  MRTFB  is  the 
accurate  determination  of  time  space-position 
information  (TSPI).  The  position  (and  some- 
times velocity  and  acceleration)  of  a test  object 
(e.g.,  aircraft,  missile,  ship)  is  measured  by 
TSPI  instrumentation  as  a function  of  time  for 
use  in  test  reconstruction  and  evaluation,  and 
for  range  safety.  Advances  in  weapon  system 
capabilities,  increasing  requirements  for  more 
realistic  testing/training,  and  growing  con- 
straints imposed  by  civil  encroachment  on  test 


ranges  have  combined  to  motivate  development 
of  new  time  space-position  information 
systems. 

The  Extended  Area  Test  System  under 
development  at  the  Pacific  Missile  Test  Center, 
Pt.  Mugu,  California,  will  provide  TSPI, 
communications  and  target  drone  control  capa- 
bilities in  the  PMTC  Outer  Sea  Test  Range  up 
to  several  hundred  nautical  miles  from  land. 
See  Figure  1.  The  movement  of  test  operations 
seaward  is  necessitated  by  the  higher  dynamics 
of  current  and  planned  weapon  systems  and  by 
the  encroachment  of  Outer  Continental  Shelf 
oil  exploration  on  traditional  PMTC  operating 
areas  closer  to  shore.  The  Extended  Area  Test 
System  will  accommodate  nearly  100  test  ob- 
jects and  will  use  equipment  modules  aboard 
test  vehicles,  processing  and  relay  subsystems 
aboard  special  instrumented  aircraft,  and 
ground  based  reference  and  computation 
systems. 

At  the  Hill/Wendover/Dugway  complex 
in  Utah,  a High  Accuracy  Multiple  Object 
Tracking  System  is  being  installed,  principally 
for  remotely  piloted  vehicle  testing.  This  equip- 
ment can  track  more  than  20  airborne  and 
surface  vehicles  simultaneously  and  display  the 
vehicle  positions  in  real  time. 

The  Air  Combat  Maneuvering  Range  and 
Air  Combat  Maneuvering  Instrumentation  are 
related  TSPI  systems  developed  by  the  Navy 
and  Air  Force  for  use  during  operational  test 
and  evaluation  and  training  exercises.  See  Fig- 
ure 2.  The  systems  permit  real  time  evaluation 
of  tactics  and  simulated  weapon  firings. 
Recording,  playback  and  display  features  allow 
postflight  analysis  of  maneuvers. 

To  provide  an  “at-sea”  environment  for 
operational  test  and  evaluation  and  training, 
the  Navy  is  developing  the  Mobile  Sea  Range. 
See  Figure  3.  When  fully  implemented  two 
mobile  sea  range  systems  will  furnish  a port- 
able, shipboard  time  space-position  capability 
for  Atlantic  and  Pacific  operations.  Divorced 
from  direct  support  by  land  based  elements,  the 
mobile  sea  range  will  permit  tracking,  commu- 
nications, and  related  functions  for  large  num- 
bers of  airborne,  surface  and,  eventually,  sub- 
surface participants. 
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Figure  1.  Extended  Area  Test  System 
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Figure  2.  Air  Combat  Maneuvering  Range 


Vot  I 


No.  s 


65 


i 


! 


Figure  3.  A Typical  Mobile  Sea  Range 


range  safety,  and  test  evaluation.  At  the  Naval 
Air  Test  Center,  Patuxent  River,  Maryland,  the 
Real-Time  Telemetry  Processing  System  is 
being  expanded  to  accommodate  two  addi- 
tional data  channels.  This  expansion  will  enable 
the  system  to  handle  workload  requirements  of 
increasing  complexity  and  amount.  The  Air 
Force  Flight  Test  Center,  Edwards  Air  Force 
Base,  California,  is  upgrading  the  capability 
and  capacity  of  the  Automated  Flight  Test 
Data  System. 

Underwater  Instrumentation 


I To  meet  special  time  space-position  infor- 

f mation  (TSPI)  requirements  the  Navy  is  devel- 

oping and  evaluating  for  test  range  purposes  a 
Multiple-Object  Instrumentation  Radar  (MIR) 
and  a dual-frequency  tracking  radar.  The  low 
cost  Multiple-Object  Instrumentation  Radar 
will  employ  phased  array  technology  to  obtain 
higher  precision  simultaneous  TSPI  on  several 
test  vehicles  from  a single  station.  The  dual 
frequency  tracking  radar  is  designed  to  solve 
the  difficult  problem  of  accurately  tracking  a 
test  object  at  low  elevation  angles  during  low- 
altitude  or  near-horizon  operations.  The  Wide 
Area  Active  Surveillance  system  provides  for 
TSPI  on  multiple  sea  and  air  targets  in  addition 
to  performing  a surveillance  function. 

Telemetry  and  Test  Data  Processing 

Projects  are  under  way  to  improve  range 
capabilities  to  acquire,  process,  record,  display, 
and  evaluate  telemetry  test  data.  At  the  Pacific 
Missile  Test  Center,  a Computer  Centralization 
and  Modernization  Program  wil  replace  ten  ob- 
solescent computers.  The  replacement  will  be  a 
single,  integrated,  primarily  real-time  system 
built  around  a redundant  pair  of  modem  cen- 
tral processors.  At  the  Space  and  Missile  Test 
Center,  Vandenberg  Air  Force  Base,  Califor- 
nia, a Telemetry  Integrated  Processing  System 
will  replace  fourteen  computers  with  six  tele- 
metry preprocessors  connected  to  a single  cen- 
tral processing  unit. 

The  telemetry  preprocessors  will  provide 
an  enhanced  capability  to  process  data  for  mis- 
sile checkout,  calibration,  go/no-go  decisions, 


L. 


Advances  in  the  range  and  sophistication 
of  underwater  and  antisubmarine  warfare 
weapon  systems  have  created  requirements  for 
improved  instrumentation.  At  the  Navy’s  At- 
lantic .Undersea  Test  and  Evaluation  Center, 
Andros  Island,  Bahamas,  an  Advanced  Sonar 
Calibration  Range  is  being  designed  and  devel- 
oped. This  mobile  facility  will  provide  a major 
capability  for  underwater  acoustical  measure- 
ment required  by  modern  submarines  and  sur- 
face ships.  The  Barking  Sands  Underwater 
Range  Extension  project  at  the  Pacific  Missile 
Test  Center,  Hawaii  will  increase  precise  un- 
derwater test  vehicle  tracking  and  control.  This 
coverage  will  permit  more  realistic  system  test- 
ing and  antisubmarine  warfare  training  than 
are  possible  elsewhere. 

Surveillance 

Test  range  operations  with  high  perfor- 
mance aircraft,  missiles,  and  target  drones  cre- 
ate hazards  to  personnel  and  equipment.  To 
maximize  the  safety  of  participants  and  elimi- 
nate risks  for  nonparticipants  close  surveillance 
of  the  range  operating  areas  is  maintained. 


At  the  Navy’s  Atlantic  Fleet  Weapons 
Training  Facility,  Roosevelt  Roads,  Puerto 
Rico,  testing  and  training  operations  are  con- 
ducted that  involve  multiple  aircraft,  ships,  and 
remotely  controlled  target  drones,  both  air- 
borne and  surface.  Live  air-to-surface,  air-to-air 
and  surface-to-air  munitions  are  employed.  The 
location  of  this  facility  in  an  area  of  dense 
commercial  and  pleasure  ship  and  aircraft  traf- 
fic contributes  greatly  to  the  need  for  range 
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surveillance.  The  task  is  currently  accom- 
plished using  a low  data  rate  air  traffic  control 
radar  system:  A Wide  Area  Active  Surveillance 
system  is  proposed.  This  system  would  employ 
phased-array  technology  and  advanced  data 
processing  techniques  to  survey  operating  areas 
and  provide  simultaneous  high-accuracy  track- 
ing data  on  multiple  test  vehicles. 

The  R-2508  restricted  airspace  complex 
over  southern  California  provides  an  area  for 
operations  by  several  DOD  activities.  Commer- 
cial airline  traffic  crisscrosses  the  area.  Private 
aircraft  operating  under  visual  flight  rule 
(VFR)  conditions  can  appear  anywhere  at  any- 
time. To  meet  the  need  for  surveillance  and 
control  of  high  performance  military  aircraft, 
cruise  missiles  and  air-launched  ordnance,  and 
to  prevent  catastrophic  interactions  with  non- 
participating aircraft,  a joint  Army /Navy/ Air 
Force/FAA  R-2508  Enhancement  Plan  is  un- 
der implementation. 

Simulation 

The  accurate  simulation  of  the  physical 
environment  in  which  a weapon  system  must 
operate  is  crucial  in  many  test  and  evaluation 
exercises.  Valid  testing  of  aircraft  vulnerability 
to  nuclear  blasts  must  include  exposure  of 
avionics/electronic  systems  to  electromagnetic 
pulse  phenomena.  To  this  end,  the  Air  Force  is 
constructing  the  TRESTLE  facility  at  Kirtland 
Air  Force  Base,  New  Mexico,  to  produce  repre- 
sentative electromagnetic  pulse  phenomena  by 
nonnuclear  means. 

Meaningful  test  and  evaluation  of  elec- 
tronic countermeasure  equipment  and  tactics 
requires  a realistic  simulation  of  the  signals 
emitted  by  threat  systems.  At  the  Air  Force 
Tactical  Fighter  Weapons  Center  and  the  Na- 
val Weapons  Center’s  ECHO  Range,  electroni- 
cally accurate  replicas  of  enemy  land  and  ship- 
bome  anti-aircraft  missile  and  gun  systems 
have  been  constructed  to  provide  an  authentic 
environment.  These  facilities  are  expanded  and 
updated  based  on  the  most  recent  intelligence 
information. 


Mobile  Platforms 

Requirements  to  provide  tracking,  teleme- 
try, communications,  and  related  support  to 
strategic  and  tactical  missile  test  operations 
over  large  areas  remote  from  land  stations  re- 
sulted in  a fleet  of  instrumented  ships  and  air- 
craft. See  Figures  4 and  5.  The  fleet  includes  the 
Ec-13S  Advanced  Range  Instrumentation  Air- 
craft, the  EC- 121  instrumented  aircraft,  the 
Advanced  Range  Instrumentation  Ships 
(USNS  ARNOLD  and  USNS  VANDEN- 
BERG),  and  the  instrumented  ships  USNS 
WHEELING  and  USNS  REDSTONE.  These 
platforms  are  the  objects  of  continuing  im- 
provement programs  to  overcome  capability 
deficiencies.  Obsolete  and  inefficient  ships  and 
aircraft  are  retired.  Instrumented  platform 
needs  for  the  1980’s  and  beyond  are  underway. 


Figure  5.  Instniinented  Aircraft 

Electro-Optical  Systems 

The  rapid  development  of  high  energy  la- 
ser technology  has  left  the  DOD  unprepar^  to 
carry  out  a complete  test  program  in  this  area. 
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To  meet  this  deficiency  a tri-Service  study  panel 
under  DUSDR&E(T&E)  guidance  was  formed 
to  examine  joint  requirements  and  potential  ap- 
proaches for  developing  a High  Energy  Lasar 
Systems  Test  Facility  (HELSTF).  After  exam- 
ining six  alternative  sites  within  the  MRTFB 
the  panel  selected  White  Sands  Missile  Range, 
New  Mexico.  Planning  for  HELSTF  is  present- 
ly underway  with  maximum  use  being  made  of 
existing  buildings  and  instrumentation. 

In  addition,  to  accommodate  lower  energy 
lasar  seeker  tests  not  appropriate  for  the  large 
outdoor  range  environment  of  HELSTF  an 
Electro-Optical  Test  Facility  is  being  developed 
at  the  Armament  Development  and  Test  Cen- 
ter, Eglin  Air  Force  Base,  Florida. 

Propulsion  Systems 

New  levels  of  aircraft  propulsion  capabil- 
ity and  the  materials  and  techniques  by  which 
they  are  achieved  have  outdistanced  the  corre- 
sponding capacity  of  DOD  and  industrial  test 
resources.  Problems  of  currently  operating  US 
military  aircraft  such  as  compressor  stalls  and 
high  fuel  consumption  might  have  been  avoided 
had  an  adequate  test  capability  been  available. 
Construction  of  the  Aeropropulsion  Systems 
Test  Facility,  Air  Force  Arnold  Engineering 
Development  Center,  Tullahoma,  Tennessee, 
now  in  progress,  will  alleviate  this  deficiency. 
See  Figure  6.  The  Aeropropulsion  Facility,  a 
$437  million  military  construction  project,  will 
permit  ground  simulation  of  critical  aerody- 
namic parameters  so  that  propulsion  systems 
can  be  optimized  at  the  design  stage. 

Targets  and  Target  Control 

The  ultimate  noncombat  test  of  most 
weapon  systems  is  an  ability  to  inflict  damage 
on  a realistic  replica  of  the  design  threat.  Tar- 
gets realistic  in  performance,  signature,  and 
tactical  operation  are  an  important  part  of  the 
test  resource  inventory.  Two  target  systems 
now  in  development  are  typical  of  required 
threat  simulation  capability.  One  is  the  Army’s 
STREAKER  subscale  aerial  target  that  shows 
potential  for  tri-Service  use.  The  second  is  the 
Anti-Ship  Missile  Target  under  development  by 


the  Navy.  This  antiship  missile  target  is  to 
simulate  the  performance  and  signature  of  So- 
viet antiship  missiles. 

To  address  operational  deficiencies  in  tar- 
get control  capability,  an  Integrated  Target 
Control  System  was  developed  to  meet  tri- 
Service  needs  across  the  family  of  aerial  targets. 
One  of  these  needs  was  to  follow  the  DOD 
direction  to  abandon  the  existing  drone  control 
frequency  band  and  select  another  frequency 
band.  For  situations  where  threat  formation  is 
significant  (i.e.,  number  of  threat  vehicles  and 
the  spacing  and  formation  structure)  the 
Army’s  Drone  Formation  Control  System, 
White  Sands  Missile  Range,  will  provide  a 
multiple  drone  control  capability. 

Transfers  of  Resources 

Not  all  test  resource  deficiencies  are  met  by 
new  developments  and  procurements.  Often  in- 
strumentation excess  to  needs  at  one  test  range 
is  transferred  to  meet  a requirement  elsewhere 
within  the  MRTFB.  Example:  a TPQ-18  track- 
ing radar  and  two  splash  detection  radars  made 
surplus  by  the  disestablishment  of  the  Phoenix 
Islands  tracking  station  were  relocated  to 
Kwajalein  Missile  Range  and  are  operational. 
Tracking  subsystems  of  the  phased-out  NIKE- 
HERCULES  air  defense  systems  are  finding 
applications  at  several  locations  within  the 
MRTFB.  Other  cooperative  ventures  include 
the  operation  by  NASA  of  the  Air  Force  FPS- 
16  radar  at  Kokee  Park,  Hawaii,  in  support  of 
Navy  programs  and  the  use  of  the  NASA  track- 
ing facility.  Wallops  Island,  Virginia,  by  the 
Naval  Air  Test  Center,  Patuxent  River,  Mary- 
land. 

Encroachment 

The  adequacy  of  DOD  test  facilities  in- 
volves the  provision  of  sufficient  land,  sea, 
airspace,  and  electromagnetic  environment  in 
which  to  conduct  test  and  evaluation.  The  De- 
partment of  Defense  currently  has  significant 
encroachment  problems  from  sources  such  as 
energy  exploration  or  exploitation,  commercial 
and  private  aviation  and  marine  activity,  civil 
demands  for  access  to  or  joint  use  of  real  estate. 
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Figure  6.  US  Air  Force  Arnold  Engineering  Development  Center 


electromagnetic  emission,  and  international 
pressures  (political  and  financial)  on  our  for- 
eign operating  locations.  At  many  facilities, 
these  encroachment  areas  are  present  in  various 
combinations  further  intensifying  the  problem. 
Major  problems  over  the  past  few  years  have  in- 
volved airspace  usage  by  both  the  military  and 
civilian  aviation  sectors  and  oil  exploration/ex- 
ploitation activities  on  the  Outer  Continental 
Shelf 

R-2508  Restricted  Airspace 

A serious  airspace  encroachment  problem 
involves  the  R-2508  Restricted  Airspace  Com- 
plex located  in  the  Upper  Mojave  Desert  of 
California.  The  complex  includes:  the  Naval 
Weapons  Center,  China  Lake;  the  Air  Force 
Flight  Test  Center,  Edwards  Air  Force  Base; 
George  Air  Force  Base;  and  the  Army’s  Fort 
Irwin.  The  R-2508  Complex  is  a multibillion 
dollar  tri-Service  test  complex  used  by  DOD  for 
the  advancement  of  weapons  systems  technol- 
ogy. The  principal  DOD  activities  are  aircraft 
flight  testing,  RDT&E  of  weapons,  electronic 
countermeasure  and  electronic  systems  testing, 
training  of  pilots  and  tactical  air  crews,  and 
large  scale  ground  force  exercises.  The  area  is 
roughly  140  miles  long  by  1 10  miles  wide  and 
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consists  of  a high  level  restricted  area  (R-2508) 
that  overlays  six  independently  managed  re- 
stricted areas  where  aircraft  and  other  weapons 
systems  are  tested  and  where  operational  train- 
ing is  conducted.  The  low  level  restricted  areas 
are  separated  by  unrestricted  airspace  within 
which  the  flying  public  may  operate.  Because  of 
the  mountainous  terrain  general  aviation  trafTic 
naturally  funnels  into  areas  immediately  adja- 
cent to  high  performance,  high  cost  test  activi- 
ties. These  geographical  constraints  and  the 
lack  of  adequate  surveillance  facilities  create 
conditions  for  potential  midair  collisions.  Cur- 
rently, large  areas  of  the  complex  have  neither 
radar  nor  communications  coverage  of  military 
or  civil  operations.  The  air  traffic  problem  is 
complicated  by  the  fact  that  commercial  opera- 
tions in  and  around  the  R-2508  restricted  area 
are  expanding  at  a rapid  rate.  With  plans  for 
Palmdale,  just  south  of  Edwards  Air  Force 
Base,  to  become  the  major  Los  Angeles  air 
terminal  in  the  1980-1990  time  period,  this 
expansion  will  surely  continue. 

After  detailed  analysis,  it  was  concluded 
that  expansion  of  the  present  radar  network 
and  the  FAA-operated  Radar  Approach  Con- 
trol Center  at  Awards  Air  Force  Base  would 
rectify  the  existing  situation.  A program  is  in 
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progress  to  integrate  three  long  range  radars, 
two  approach  control  radars  and  six  gap  filler 
radars.  A mosaic  (through  a computer  system 
and  associated  software  and  peripheral  equip- 
ment) is  to  provide  a capability  for  real  time 
control  of  R-2508. 

Outer  Continental  Shelf  Oil  Exploration 

In  June  1971,  the  Bureau  of  Land 
Management,  Department  of  Interior,  an- 
nounced an  extensive  plan  to  exploit  Outer 
Continental  Shelf  oil  and  natural  gas  reserves. 
Areas  off  the  coasts  of  southern  California  and 
the  Gulf  of  Mexico  were  of  particular  concern 
to  DOD  because  of  extensive  use  of  the  areas 
for  testing  weapons  and  training  operational 
forces.  In  the  Gulf  of  Mexico,  DOD  missions 
include  gunnery,  air-to-air  missile  firings,  bomb 
drops,  high-altitude  probes,  electronic  warfare 
exercises,  and  aircraft  carrier  landing  and  take- 
off training.  In  a 70-mile  wide  corridor  south  of 
Eglin  Air  Force  Base,  of  the  640,000  acres 
proposed  for  lease,  DOD  negotiated  exclusion 
of  17,000  acres.  Off  the  coast  of  southern  Cali- 
fornia, DOD  missions  include  air-to-air,  air-to- 
ground,  and  ground-to-air  missile  test  firings  as 
well  as  extensive  fleet  training.  Of  the  7.7  mil- 
lion acres  proposed  for  lease,  DOD  negotiated 
exclusion  of  3.4  million  acres. 

UTILIZATION 

Duplication 

In  addition  to  insuring  the  adequacy  of  the 
facility  base,  it  is  necessary  to  insure  the  effi- 
cient use  of  test  resources  by: 

• Authorizing  only  those  test  resources 
that  are  needed, 

• Retiring  unused  or  obsolete  assets  and, 

• Eliminating  unnecessary  duplications. 

In  discharging  this  responsibility.  Defense 
agencies  must  demand  that  all  facilities  are 
reviewed  frequently  for  workload,  capacity, 
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and  quality-of-support  indicators  that  signal 
potential  problem  areas. 

The  Military  Departments  are  alert  to 
savings  opportunities.  The  Office  of  the  Secre- 
tary of  Defense  provides  the  thread  that  ties  all 
DOD  test  facilities  together.  Opportunities  for 
economies  through  inter-Service  use  of  re- 
sources are  identified  and  joint  studies  and 
analyses  are  conducted  to  evaluate  alternative 
support  methods.  Funds  to  maintain  and  oper- 
ate the  26  facilities  composing  the  MRTFB 
represent  a significant  portion  of  the  Defense 
budget.  The  cost  of  support,  the  complexity  of 
user  requirements,  increasing  encroachment, 
and  inflation  combine  to  reduce  capacity  and 
dollars  available  for  improvements.  These  fac- 
tors make  it  imperative  that  the  extremely  im- 
portant MRTFB  support  function  be  carried 
out  in  a cost  effective  manner,  and  that  any 
funds  and  effort  expended  on  unwarranted  du- 
plication be  redirected  to  overcome  program 
deficiencies. 

General  Accounting  Office  Duplication 
Studies 

Improved  management  of  MRTFB  func- 
tions is  of  vital  interest  to  the  Congress.  The 
General  Accounting  Office  and  DOD  were 
directed  to  take  certain  actions.  In  a report 
entitled,  “Review  of  the  Adequacy  of  Depart- 
ment of  Defense  Test  Resources,”  (dated  April 
30,  1975  and  submitted  to  the  Senate  Appropri- 
ations Committee),  the  General  Accounting 
Office  concluded  that  a “high  degree  of  dupli- 
cation in  test  capabilities  appears  to  exist.”  In 
May  1975,  the  Senate  Appropriations  Commit- 
tee directed  that  General  Accounting  Office 
study  the  specific  areas  of  ordnance,  propul- 
sion, and  underwater  testing  to  determine  the 
extent  of  duplication  and  whether  such  duplica- 
tion was  warranted.  The  GAO  report  of  March 
1,  1976,  titled,  “Does  the  Department  of  De- 
fense Have  More  Test  Capacity  Than  It 
Needs?”  concluded  that  potential  capacity  ex- 
ceeded workload  in  the  ordnance  and  engine 
test  cell  areas  and  implied  that  the  workload 
could  be  accomplished  at  fewer  facilities.  The 
General  Accounting  Office  report  substan- 
tiated the  need  for  the  existing  family  of  under- 
water test  facilities. 
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Department  of  Defense  Utilization 
Studies 

The  Office  of  the  Secretary  of  Defense 
issued  directives  to  the  Services  in  1976  to 
review  workload,  capacity  and  management  of 
several  activities  involved  in  supporting  ord- 
nance, propulsion,  ballistic  missile,  parachute 
and  Remotely  Piloted  Vehicle  testing.  The  in- 
tent was  to  cause  consolidation  of  workload 
and/or  management  leading  to  eventual  reduc- 
tion or  closure  of  specific  test  support  facilities. 
This  action,  still  in  process,  is  a precursor  of 
further  activity  of  this  type.  The  requirements 
for  test  support  change  with  national  priorities. 

Uniform  Measures 

The  availability  of  meaningful  MRTFB 
workload  and  capacity  data  in  uniform  units  is 
a prerequisite  to  the  evaluation  of  efficiency, 
duplication,  and/or  under-utilized  resources. 
Existing  data  has  lacked  the  detail  and  unifor- 
mity needed  for  significant  analysis.  In  its  1976 
study  of  DOD  test  facilities,  the  GAO  had  diffi- 
culty in  measuring  workload  and  capacity  and 
concluded  that  "the  measurement  of  test  facili- 
ty workload  and  capacity  is  very  complex  and 
no  common  criteria  exists  for  uniform  measure- 
ment of  test  facility  capacity  and  use.”  The 
OSD  agreed  and  is  working  on  a system  that  is 
to  accurately  measure  test  facility  workload,  ca- 
pacity and  capability  in  uniform  units. 

In  1975  an  effort  was  initiated  to  evaluate 
techniques  for  the  uniform  collection,  mainte- 
nance, and  use  of  test  facility  data  for  the 
purpose  of  identifying  unwarranted  duplica- 
tion. Early  in  the  program,  broader  applicabil- 
ity of  the  data  was  recognized.  The  data  pro- 
vided resource  identification,  capability,  work- 
load, and  capacity.  The  resultant  system  was 
implemented  on  a trial  basis  at  three  MRTFB 
facilities  (Yuma  Proving  Ground,  Pacific  Mis- 
sile Test  Center  and  Arnold  Engineering  Devel- 
opment Center)  for  a 3 month  period  in  1976. 
To  test  the  usefulness  of  the  data,  the  three 
facilities  were  asked  to  submit  workload  and 
capacity  data  in  the  new  format  in  support  of 
the  FY  1978  budget  submission.  The  system  is 
based  on  using  investment  resources  as  the 
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common  baseline  with  data  on  capability,  ca- 
pacity, workload  and  utilization  being  accu- 
mulated and  reported  for  each  resource.  These 
data  can  then  be  aggregated  by  program  sup- 
ported, time  period  and  event  to  provide  the 
total  usage  relative  to  capacity  and  application. 
The  Military  Departments  are  now  completing 
the  evaluations. 

Cost  Savings 

In  the  past  several  years,  OSD  and  the  Ser- 
vices have  had  an  active  program  to  retire  obso- 
lete instrumentation  and  equipment,  to  close  fa- 
cilities that  are  no  longer  required,  and  to  real- 
locate underused  resources.  From  FY  1970 
through  FY  1975  major  resources  and  facilities 
were  deactivated.  The  result  was  an  annual  sav- 
ings in  FY  1976  and  subsequent  years  of  $28.1 
million.  Specific  actions  consisted  of  deacti- 
vating seven  ships  (TWIN  FALLS,  WATER- 
TOWN,  RANGE  TRACKER,  HUNTS- 
VILLE, SUNNYVALE,  LONGVIEW,  MER- 
CURY) saving  $20.5  million,  three  test  support 
locations  (Eniwetok,  Eleuthera,  Green  River) 
saving  $1.8  million,  three  major  radar  installa- 
tions (AMRAD,  HAPDAR,  SIMPAR)  saving 
$2.4  million,  and  releasing  10  test  support  air- 
craft saving  $3.4  million.  In  addition,  $27  mil- 
lion in  cost  avoidance  was  achieved  by  transfer- 
ring seven  major  radars. 

The  Senate  Committee  on  Appropriations 
Report  94—446,  dated  November  6,  1975,  di- 
rected DOD  to  effect  savings  in  test  ranges  and 
test  facilities  in  FY  1976  and  197T,  by  consoli- 
dations and/or  improved  management,  and  to 
reflect  these  savings  in  the  FY  1977  budget. 
Several  actions  were  taken  in  FY  1976  resulting 
in  annual  savings  of  $15.9  million  in  FY  1977 
and  beyond.  The  Phoenix  Islands  instrumenta- 
tion complex  was  reduced  to  caretaker  status 
saving  $8.7  million^  the  Air  Force  Special 
Weapons  Center  was  deactivated  saving  $4.5 
million,  and  1 1 more  test  support  aircraft  were 
released  saving  $2.7  million.  In  addition,  $18 
million  in  cost  avoidance  was  achieved  by  plac- 
ing the  WHEELING  and  VANDENBERG 
ships  in  ready  reserve  for  14  months  ($8.9 
million),  consolidating  the  Army  Test  Boards 
($0.7  million)  and  transferring  three  more  ra- 
dars ($8.4  million).  In  FY  1977  six  more  test 
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support  aircraft  are  being  released  for  an  annual 
saving  of  $1.8  million,  and  a reduction  of  about 
500  civilian  manpower  positions  is  being 
achieved  among  several  test  facilities,  saving 
$9.6  million. 


Since  FY  1970  a total  annual  cost  savings 
of  $55.4  million  has  been  realized  and  is  re- 
flected in  the  FY  1978  budget.  Reallocated 
MRTFB  resources  have  resulted  in  one  time 
savings  of  $45.0  million.  Cost  escalations  in 
personnel,  contracts,  materials,  supplies,  and 


i- 


Mr.  John  W.  McCord  is 
the  Director  for  Test  Re- 
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CA;  and  Assistant  Director  and  Technical  Assistant  for 
Space  and  Astronautics,  Office  of  the  Chief  of  Naval  Op- 
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Mr.  McCord,  following  his  naval  service,  was  a tech- 
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berg  AFB,  CA.  Later  as  an  Air  Force  civilian  Mr. 
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in  Physics  from  the  University  of  California  at  Berkley. 


energy  have  more  than  offset  the  savings,  mak- 
ing savings  visibility  difficult.  An  active  pro- 
gram of  retiring  obsolete  or  underused  re- 
sources and  closing  sites  when  possible  will 
continue.  Actions  already  accomplished  or  un- 
derway have  reduced  the  facilities  to  a mini- 
mum configuration.  To  achieve  savings  in  the 
future,  we  must  took  toward  significant  alterna- 
tives such  as  closure  of  major  facilities.  Such 
actions  would  allow  the  Services  to  strengthen 
the  remaining  facilities  and  provide  the  re- 
sources needed  to  support  the  test  and  evalua- 
tion of  ongoing  and  future  Defense  programs. 
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TEST  AND  EVALUATION  OF 
TRI-TAC  EQUIPMENT 

by 

Lt  Col  David  L.  Washington,  Department  of  Air  Force 


In  February  1971,  the  Joint  Tactical  Communication  (TRI-TAC)  Program  was  chartered 
primarily  to  insure: 

The  necessary  degree  of  interoperability  among  tactical  communications 
systems  and  other  DOD  telecommunications  systems. 

Timely  fielding  of  new  tactical  communication  equipment  required  by  the 
Armed  Forces  to  perform  their  mission  reflecting  the  most  effective 
technologies. 

Elimination  of  duplication  in  the  development  of  Military  Service 
equipment.' 

TRI-TAC  developments  including  all  trunking,  access  and  switching  equipment  for  mobile 
and  transportable  tactical  multichannel  systems,  including  associated  systems  control  and 
technical  control  facilities,  local  distribution  equipment,  voice,  teletype,  data  and  ancillary 
terminal  devices  and  associated  communications  security  equipment. 


INTRODUCTION 

From  the  commencement  of  the  TRI- 
TAC  Program,  it  was  envisioned  that  test  and 
evaluation  would  be  a joint  endeavor  conducted 
by  the  military  departments  and  DOD  agencies 
under  the  “Lead  Service”  concept.  In  recogni- 
tion of  the  complexity  of  such  an  undertaking,  a 
comprehensive  joint  test  program  was  estab- 
lish^. The  program  was  to  provide  assurance 
that  the  objectives  of  equipment/system 
commonalty,  compatibility,  and  interoperabil- 
ity would  be  maintained  in  the  most  cost  effec- 
tive manner.  Moreover,  the  TRI-TAC  spon- 
sored equipment,  engineered  primarily  for  digi- 
tal applications,  must  interoperate  with  an  ex- 
tensive existing  inventory  of  primarily  analog 
equipment,  thus  adding  to  the  difficulties  of  an 
already  complex  test  and  evaluation  task. 


The  joint  test  and  evaluation  program  is  a 
“new  way  of  doing  business”  in  the  test  and 
evaluation  arena  for  tactical  communications. 
The  dominant  characteristic  of  a joint  test  pro- 
gram is  its  provisions  for  participating  services 
and  agencies  to  jointly  plan  and  conduct  testing 
with  TRI-TAC  Office  coordination.  To  achieve 
program  objectives  of  interoperability  and  com- 
patibility, it  is  imperative  that  a comprehensive 
joint  performance  be  realized  from  the  military 
departments  and  DOD  agencies;  that  is,  the 
TRI-TAC  Program  participants.  Accordingly, 
an  approach  encompassing  joint  definitions  of 
policies  and  responsibilities  has  been  estab- 
lished to  implement  the  test  and  evaluation 
program.' 


T&E  CONCEPTS  AND  REQUIREMENTS 

Primarily,  the  TRI-TAC  Joint  Test  and 
Evaluation  Program  is  governed  by  the  DOD 
process  for  the  acquisition  of  defense  systems. 
The  acquisition  program  process  is  divided  into 
four  phases.  Program  decisions  are  based  on 
evaluated  test  results  for  each  phase  before  the 
decision  to  proceed  to  the  subsequent  phase  is 
made.’  Each  phase  has  a designated  milestone 
for  the  decisionmaker  and  an  associated 
information-bearing  testing  concept.’ 

• Milestone  0 — Program  initiation  (con- 
ceptual phase) 

• Milestone  I — Demonstration  and  vali- 
dation (validation  phase) 

• Milestone  //-Full-scale  engineering 
development  (full-scale  engineering  de- 
velopment phase) 

• Milestone  III — Production/Deploy- 
ment (production  and  development 
phase) 

The  concept  for  joint  test  and  evaluation 
was  formulated  based  on  the  framework  of  the 
policies  and  procedures  established  by  the  Serv- 
ices and  Agencies  for  acquiring  heretofore 
unique  equipments.  From  this  framework,  the 
TRI-TAC  Office  developed  common  test  terms 
for  each  test  phase  to  be  used  in  the  joint  test  of 
TRI-TAC  sponsored  acquisitions.  The  follow- 
ing definitions  of  test  periods  and  associated 
activities  were  promulgated  as  DOD  guidance 
to  the  Services  and  Agencies.’ 

Conceptual  Phase  - The  conceptual  phase 
extends  from  the  recognition  of  a needed  opera- 
tional capability  to  the  program  decision  (Mile- 
stone I)  where  initiation  of  the  validation  phase 
is  authorized.  Test  planning,  if  required  during 
the  conceptual  phase,  is  the  responsibility  of  the 
acquisition  Service  in  coordination  with  other 
program  participants.  Test  plans  are  designed 
to  demonstrate  the  feasibility  of  a proposed 
acquisition  program  providing  a desired  opera- 
tional capability  or  solving  a military  problem. 


Tests  during  this  phase  are  designated  as  re- 
search tests  or  feasibility  tests.  Both  Govern- 
ment and  contractor  evaluations  of  test  results 
support  a decision  to  proceed  to  the  validation 
phase. 

Validation  Phase  - The  validation  phase 
commences  after  an  affirmative  program  deci- 
sion (Milestone  I)  and  continues  through  the 
ratification  decision  point,  (Milestone  II), 
where  full-scale  engineering  development  is  au- 
thorized. Contractor  testing  during  the  valida- 
tion phase  is  identified  as  contractor  demon- 
stration. Contractor  demonstration  plans  and 
procedures  are  approved  by  the  Service  charged 
with  acquisition.  Contractor  demonstration 
plans  are  intended  to  insure  the  verification  of 
preliminary  design  and  engineering,  and  to 
demonstrate  the  degree  of  compliance  with 
specifications  and  requirements.  Government 
testing  in  this  phase  is  identified  as  development 
suitability  tests.  Test  coordination  is  the 
responsibility  of  the  acquiring  Service  program 
participants.  Development  suitability  tests  are 
performed  to  demonstrate  that  technical  and 
operational  risks  have  been  identified  and  that 
solutions  are  achievable. 

Full-Scale  Engineering  Development 
Phase  - The  full-scale  engineering  development 
phase  extends  from  the  ratification  decision 
point  (Milestone  II)  to  the  production  decision 
point  (Milestone  III).  The  full-scale  engineer- 
ing development  phase  is  directed  toward  con- 
firming that 

• engineering  is  complete, 

• previously  identified  technical  uncer- 
tainties have  been  resolved, 

• the  operational  suitability  of  the  item 
has  been  determined,  and 

• realism  of  the  plan  has  been  confirmed. 

Tests  during  this  phase  are  contractor  develop- 
ment tests  and  Government  tests  consisting  of 
Development  Test  and  Evaluation  (developer- 
oriented)  and  Initial  Operational  Test  and  Eval- 
uation (user-oriented). 
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Production  and  Deployment  Phase  - The 

production  and  deployment  phase  commences 
after  the  production  decision.  In  this  phase,  the 
equipment/system  is  produced  and  delivered  as 
an  effective  operational  and  supportable  com- 
modity for  the  users.  Testing  during  production 
is  categorized  as  Production  Acceptance  Test 
and  Evaluation  (Contractor)  and  Operational 
Test  and  Evaluation  (Government). 

As  can  be  seen  from  the  complexity  and 
magnitude  of  test  and  evaluation  associated 
with  TRI-TAC  sponsored  equipment,  an  effi- 
cient and  economical  management  structure 
had  to  be  designed  to  provide  an  effective  sys- 
tem for  coordinating  and/or  managing  all 
phases  of  test  and  evaluation. 

MANAGEMENT  AND 
ORGANIZATION 

While  it  was  initially  envisioned  that  each 
Service  tasked  with  acquiring  an  individual 
item  of  TRI-TAC  equipment  would  be  desig- 
nated “Lead  Service”  for  test  and  evaluation,  it 
became  apparent  that  considering  factors  such 
as  timeliness,  cost  effectiveness,  schedule  con- 
straints, long  term  test  viability  relative  to  con- 
figuration control,  and  test  repeatability  testing 
efforts  precluded  multiple  assignments  of  con- 
current “Lead  Service.”  Accordingly,  theTRI- 
T AC  Office  provided  the  Office  of  the  Secretary 
of  Defense  with  an  indepth  study  of  short  and 
long  range  testing  applications  resulting  from 
the  introduction  of  modern  digital  state-of-the- 
art  communications  equipment  to  the  Service 
inventories.*  The  TRI-TAC  study  recom- 
mended the  present  concept  of  establishing  a 
permanent  TRI-TAC  Joint  Test  Organization 
and  joint  test  facility  for  achieving  test  and 
evaluation  for  TRI-TAC  sponsored  equipment. 
The  recommendation  was  approved.  With  the 
exception  of  the  permanent  OSD-cadre  of  the 
Joint  Test  Organization,  a special  organiza- 
tional structure  is  not  required  to  support  the 
TRI-TAC  test  and  evaluation  program.  How- 
ever, there  are  other  organizational  activities 
that  enter  into  the  test  and  evaluation  arena. 
See  Figure  1.  Present  responsibilities  of  the  test 
and  evaluation  program  participants  are  given 
here.' 
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The  Assistant  Secretary  of  Defense  for 
Communications,  Command  and  Control  and 
Intelligence  (ASD  (CH ) provides  DOD  staff 
supervision  over  the  TRI-TAC  Program  and 
insures  the  active  participation  of  relevant 
DOD  components  in  test  and  evaluation 
activities. 


The  Deputy  Undersecretary  of  Defense 
Research  and  Engineering,  (Test  and  Evalua- 
tion) DUSDR&E(T&E)  provides  DOD  Staff 
monitoring  of  the  planning  and  conduct  of  test 
and  evaluation  and  makes  an  independent  eval- 
uation on  the  adequacy  of  testing. 


The  Joint  Chiefs  of  Staffmoniior  the  plan- 
ning and  conduct  of  test  and  evaluation  to 
insure  that  joint  requirements  are  addressed, 
and  direct  the  accomplishment  of  follow-on 
joint  operational  test  and  evaluation. 


The  Joint  Tactical  Communications 
(TRI-TAC)  Office responsible  for  the  overall 
coordination  of  TRI-TAC  Program  joint  test 
and  evaluation.  The  joint  test  element  and  joint 
test  facility  activities  are  established  and 
managed  by  this  office.  This  office  provides  an 
independent  evaluation  of  test  results  to 
ASDfC^I). 


The  Joint  Test  Coordinating  Committee  s 
composed  of  senior  Service/ Agency  representa- 
tives and  is  the  principal  focal  point  for  the 
monitoring  of  all  aspects  of  the  test  and  evalua- 
tion program.  The  Committee  arbitrates  prob- 
lems and  advises  the  Director,  TRI-TAC  Of- 
fice, on  progress  of  test  and  evaluation 
activities. 


The  Services  AnecX  the  conduct  of  develop- 
ment test  and  evaluation  and  initial  operational 
test  and  evaluation;  participate  actively  in  all 
aspects  of  test  and  evaluation;  and  provide  an 
independent  evaluation  to  the  ASD(C^I)  and 
the  Joint  Chiefs  of  Staff  on  the  results  of  testing. 
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National  Security  Agency  participates  in 
communications  security  aspects  of  the  test  and 
evaluation  program.  This  Agency  indepen- 
dently evaluates  test  results  with  respect  to  the 
security  of  the  equipment  or  system  under  test. 

The  Defense  Communications  Agency 
participates  appropriately  in  all  aspects  of  test 
and  evaluation  and  independently  evaluates  test 
results  with  respect  to  the  suitability  of  the 
equipment  under  test  for  use  in  the  Defense 
Communications  Agency, 

In  support  of  the  test  and  evaluation  effort, 
a Joint  Test  Organization  was  established.’  The 
Joint  Test  Organization  consists  of  the  joint  test 
element,  facility  support  element,  test  support 
element.  Service/ Agency  test  officer  for  devel- 
opment test  and  evaluation.  Service  test  officers 
for  initial  operational  test  and  evaluation,  and 
liaison  officers.  See  Figure  2.  The  Joint  Test 
Organization  is  to  serve,  primarily  as  the  focal 
point  for  the  planning,  conducting,  and  report- 
ing of  government  testing.  The  Joint  Test  Orga- 
nization, with  the  exception  of  the  joint  test 
element,  is  composed  of  resources  from  the 
Services/ Agencies. 

The  Joint  Test  Element  is  a permanent 
DOD  staff  reporting  directly  to  the  Director, 
TRI-TAC  Office.  The  joint  test  element  has  the 
responsibility  to  coordinate  the  activities  of  the 
Joint  Test  Organization  and  provides  the  per- 
manent personnel  required  to  support  testing. 


The  Facility  Support  Element  consists  of 
inventory  equipment  and  personnel  resources 
required  to  maintain  the  system  baseline  at  the 
test  facility  on  a continuing  basis.  The  Army 
will  provide  the  facility  support  element. 


The  Test  Support  Element  consists  of  per- 
sonnel and  equipment  resources  keyed  to  the 
execution  of  a specific  test  plan  for  an  individ- 
ual item  of  equipment.  There  will  be  numerous 
test  support  elements  throughout  the  test  and 
evaluation  program.  These  elements  will  be 
phased  in  and  out  of  the  test  facility  as  required. 
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The  Service/Agency  Test  Officers  (de- 
velopment test  and  evaluation  and  initial  opera- 
tional test  and  evaluation  officers)  are  person- 
nel from  the  Services  who  act  as  on-site  execu- 
tive agents  for  the  respective  Service  test 
Agencies/Commands  on  matters  relating  to 
test  and  evaluation.  These  officers  may  act  as 
test  directors  for  the  individual  Service  test 
commands. 


Liaison  Officers  are  assigned  to  the  Joint 
Test  Organization  as  required. 

The  successful  operation  of  the  Joint  Test 
Organization  will  involve  considerable  commu- 
nication, coordination  and  cooperation  among 
its  organizational  elements.  Primary  objectives 
of  the  Joint  Test  Organization  are: 

• to  insure  thai  test  and  evaluation  plan- 
ning includes  sufficient  testing  of  each 
acquisition  item  and 

• to  insure  testing  in  the  most  econom- 
ical manner  by  eliminating  duplication 
and  requiring  minimum  resources  to 
satisfy  the  desired  test  objectives. 


In  addition  to  the  normal  preparation  of 
contractor  test  plans  and  government  develop- 
ment and  operational  test  plans,  the  Joint  Test 
Organization  will  prepare  an  integrated  test 
plan.  The  integrated  test  plan  will  be  used 
primarily  as  an  OSD  management-level  docu- 
ment based  on  the  development  test  and  evalua- 
tion and  initial  operational  test  and  evaluation 
plans.  The  integrated  test  plan  will  be  used  as 
the  DOD-level  coordination  document  for  each 
major  acquisition  of  TRI-TAC  sponsored 
equipment.  Moreover,  this  plan  will  be  the 
document  that  identifies  overall  milestones, 
schedule  and  resources;  and,  accommodates  the 
requirements  of  other  joint  test  activities  con- 
ducted simultaneously  at  the  joint  test  facility. 
Accordingly,  test  and  evaluation  planning  must 
begin  early  in  the  acquisition  program  and 
encompass  both  contractor  and  government 
test  and  evaluation.*  Responsibilities  for  the 
specific  plans  are  assigned  as  outlined  here. 
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Contractor  Development  Test  Plans  - The 
contractor  prepares  the  plans  and  submits  them 
to  the  acquisition  Service/ Agency.  The  acqui- 
sition Service/Agency  is  responsible  for  coordi- 
nating the  review  of  contractor  development 
test  plans  with  the  program  participants.  After 
review  and  formal  coordination,  the  plan  is 
approved  or  disapproved  by  the  acquisition 
Service/ Agency  and  the  contracting  officer 
transmits  the  decision  to  the  contractor. 


Development  Test  and  Evaluation  and  Ini- 
tial Operational  Test  and  Evaluation  Plans — 

The  acquisition  Service/ Agency  is  responsible 
for  preparation  and  joint  coordination  of  these 
government  plans.  Development  test  and  evalu- 
ation and  initial  operational  test  and  evaluation 
plans  will  be  developed  by  joint  working  groups 
as  deemed  necessary  by  the  respective  testing 
commands  of  each  acquisition  Service.  After 
coordination  and  concurrence  by  the  program 
participants,  the  plans  are  submitted  to  the  Ser- 
vice/Agency test  officer  of  the  Joint  Test  Or- 
ganization for  distribution  within  that  organi- 
zation for  use  as  inputs  to  the  integrated  test 
plan.  The  Director,  TRI-TAC  Office,  reviews 
the  test  plans  and  monitors  the  progress  of  all 
acquisitions  to  assure  adherence  to  TRI-TAC 
Program  objectives  and  system  design  con- 
cepts. 


Integrated  Test  Plan — Integrated  test 
plans  will  be  developed  by  the  acquisition  Ser- 
vice test  officer  assisted  by  the  Service  test  offi- 
cers of  the  Joint  Test  Organization  and  the  joint 
test  element  with  appropriate  inputs  from  the 
development  test  and  evaluation  and  initial  op- 
erational test  and  evaluation  plans.  An  integra- 
ted test  plan  will  be  prepared  for  each  acquisi- 
tion under  test  and  will  address  the  integration 
aspects  of  development  test  and  evaluation  and 
initial  operational  test  and  evaluation  for  that 
acquisition.  Integrated  test  plans  are  reviewed 
and  coordinated  by  the  Services/ Agencies  prior 
to  submission  by  TRI-TAC  to  the  Undersecre- 
tary of  Defense  Research  and  Engineering  (Test 
and  Evaluation)  and  the  Joint  Chiefs  of  Staff  for 
OSD-level  review  and  coordination. 


Production  Acceptance  Test  and  Evalua- 
tion Plans  - The  contractor  prepares  the 
production  acceptance  test  and  evaluation 
plans  and  submits  them  to  the  acquisition 
Service/Agency  for  review  and  approval.  The 
acquisition  Service/ Agency  is  responsible  for 
coordinating  the  review  of  these  plans  with 
other  program  participants. 


Operational  Test  and  Evaluation  Plans — 

The  joint  operational  test  and  evaluation  plans 
will  be  developed  by  the  Services  as  directed  by 
the  Chairman  of  the  Joint  Chiefs  of  Staff. 
Follow-on  operational  test  and  evaluation  plans 
will  be  developed  as  directed  by  the  respective 
Service/ Agency.  The  Deputy  Undersecretary 
of  Defense  Research  and  Engineering  (Test  and 
Evaluation)  will  review  operational  test  and 
evaluation  plans  to  determine  plan  adequacy. 
In  cases  where  the  facilities  of  the  joint  test  fac- 
ilities will  be  used  for  operational  test  and  eval- 
uation the  Commander  of  the  joint  test  element 
will  review  plans  to  assure  adherence  to  TRI- 
TAC  Joint  Test  Program  schedules.  Joint  oper- 
ational test  evaluations  will  be  provided  to  the 
TRI-TAC  Office  indicating  the  degree  to  which 
TRI-TAC  sponsored  equipment  meets  joint  or 
Service/ Agency  operational  requirements. 


After  test  plans  have  been  developed  (con- 
currences have  been  received  and  the  plans 
completed)  the  acquisition  Service/ Agency 
must  prepare  for  executing  those  plans.  Initial 
testing  is  accomplished  at  the  contractor’s  plant 
or  at  government  approved  facilities.  The  gov- 
ernment will  witness  all  contractor  tests  to 
insure  that  the  objectives/requirements  identi- 
fied in  the  contractor  development  test  plans 
are  satisfied.  The  acquisition  Service/ Agency 
test  officers  of  the  Joint  Test  Organization  are 
responsible  for  planning  the  observance  of  con- 
tractor development  test  by  all  program  partici- 
pants and  insuring  that  evaluated  results  are 
available  for  use  at  the  next  level  of  test  plan- 
ning. These  steps  are  taken  to  eliminate  redun- 
dant testing. 

Next,  the  test  items  are  delivered  to  the 
joint  test  facility  where  the  acquisition 


Services/ Agencies  are  responsible  for  conduct- 
ing all  government  test  and  evaluation  on 
equipment  for  which  they  are  the  acquisition 
manager.  Each  acquisition  Service/ Agency  will 
designate  a test  director  to  be  responsible  for 
achieving  the  test  objectives  with  support  from 
the  other  program  participants  using  the  Joint 
Test  Organization  elements  and  the  resources 
at  the  joint  test  facility.  All  resources  will  be 
time-phased  to  insure  maximum  economy  of 
scale  throughout  the  test  period. 

The  Service/ Agency  test  officers,  assisted 
by  the  joint  test  elements,  will  prepare  and  sub- 
mit separate  development  test  and  evaluation 
and  initial  operational  test  and  evaluation  re- 
ports for  each  item  under  test.  Test  reports  will 
be  provided,  for  independent  evaluation  as  ap- 
propriate, to  all  program  participants,  to  in- 
clude the  TRI-TAC  Office,  the  Deputy  Under- 
secretary of  Defense  Research  and  Engineering 
(Test  and  Evaluation),  and  the  Joint  Chiefs  of 
Staff. 

JOINT  TEST  SITE  AND  FACILITIES 

The  joint  test  facility  is  located  at  Fort 
Huachuca,  Arizona,  with  an  adjunctive  Naval 
Telecommunications  System  test  node  at  San 
Diego,  California.  The  centralized  test  facilities 
at  Fort  Huachuca  include  a central  test  facility, 
logistics  support  test  facility,  contractor  sup- 
port facility  and  both  local  and  remote  nodes. 

Central  Test  Facility  - The  central  test 
facility  is  inclosed  by  a security  fence  and  con- 
sists of  a test  control  center  and  four  collocated 
test  nodes.  The  test  control  center  consists  of 
operating  spaces  and  facilities  required  to  con- 
trol testing.  The  heart  of  the  center  is  the  test 
control  system  that  is  designed  to  facilitate 
configuration  management  at  each  of  the  four 
nodes.  The  test  control  system  is  a semiauto- 
matic switching  and  patching  system  that  will 
be  used  to  establish  interconnections  between 
various  items  of  equipment  both  at  the  central 
test  facility  and  the  nodes.  The  test  control 
system  will  interface  with  the  main  distribution 
frame  by  means  of  audio  cable  that  carries  both 
analog  signals  and  digital  data.  The  test  control 
system  will  be  transparent  to  the  transmission 
paths  carrying  test  data. 


Three  of  the  four  local  nodes  composing 
the  central  test  facility  are  located  at  hardstand 
areas  approximately  250  feet  from  the  test  con- 
trol center.  The  fourth  node  is  collocated  with 
the  test  control  center.  Each  node  has  lighting 
and  permanent  interface  facilities  to  include 
audio,  video,  and  power  distribution  systems. 

The  nodes  will  be  configured  as  test  nodes 
representing  the  network  configurations  re- 
quired by  the  various  integrated  test  plans.  The 
separation  between  nodes  is  designed  primarily 
to  minimize  congestion  in  the  control  test  facil- 
ity and  to  reduce  the  possibility  of  electromag- 
netic interference  between  nodes. 

The  hardstand  and  permanent  interfaces 
at  each  node  are  to  facilitate  installation  and 
interconnection  of  equipment  within  each  node 
to  permit  the  rapid  establishment  of  various 
node-to-node  trunks  through  the  test  control 
center  via  the  test  control  system.  The  disposi- 
tion and  configuration  of  equipment  within 
individual  nodes  will  be  flexible  and,  except  for 
space  limitations,  independent  of  the  perma- 
nent facilities. 

Logistics  Support  Test  Facility  - The  lo- 
gistics support  test  facility  will  provide  a suit- 
able environment  for  conducting  logistics  sup- 
portability  testing,  especially  at  the  immediate 
level  of  maintenance.  Logistics  support  pack- 
ages for  equipment  under  test  will  be  stored, 
used  and  tested  in  this  facility.  The  facility  will 
provide  administrative,  work  and  storage  areas 
for  the  acquisition  Service  logistics  support  test 
and  evaluation  teams.  The  facility  provides  for 
simultaneous  accommodation  of  several  differ- 
ent test  items  during  government  testing. 

Contractor  Support  Facility  - The  con- 
tract support  facility  is  designed  to  provide 
administrative,  storage  and  maintenance  work 
areas  in  support  of  contractor  items  under  test. 

Radio  Relays  - The  radio  relays  asso- 
ciated with  the  joint  test  facility  will  serve  two 
different  functions.  First,  the  radio  relays  will 
permit  the  establishment  of  “local  node”  radio 
frequency  links  over  distances  representative  of 
actual  operational  conditions  when  such  links 
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are  required  to  provide  “electrically  realistic” 
networks.  Secondly,  during  the  deployment  of 
equipment  to  remote  operating  areas  for  initial 
operational  test  and  evaluation,  the  radio  relays 
will  be  reoriented  as  required  to  provide  radio 
frequency  links  between  the  deployed  nodes 
and  the  central  test  facility. 

Remote  Operating  Areas  - Remote  oper- 
ating areas  may  be  unimproved  areas  suitable 
for  the  deployment  of  military  personnel  and 
equipment  to  accomplish  initial  operational  test 
and  evaluation,  or  these  areas  may  be  other 
military  installations.  In  either  case,  the  areas 
must  provide  representative  environments  to 
simulate  expected  deployment  conditions.  The 
equipment  in  one  or  more  of  the  nodes  at  the 
central  test  facility  may  be  disconnected  and 
deployed  to  one  of  the  designated  remote  oper- 
ating areas.  The  distance  of  these  remote  areas 
from  the  central  test  facility  will  depend  upon 
the  desired  test  scenario. 

Naval  Telecommunications  System  Test 
Node  - In  support  of  the  joint  test  and  evalua- 
tion program  the  Navy  is  establishing,  as  an 
adjunct  to  the  TRI-TAC  joint  test  facility,  a 
Naval  Telecommunications  System  test  node  in 
the  San  Diego,  California  area.’  The  objectives 
of  the  test  node  are  to  provide  Navy  test  and 
evaluation  participation  and  support  to  the 
TRI-TAC  Program,  to  determine  the  degree  of 
interoperability  between  the  Naval  Telecom- 
munications System  and  subsystems  developed 
under  the  TRI-TAC  Program,  and  to  provide  a 
capability  for  test  and  evaluation  of  unique 
Naval  Telecommunications  equipment  during 
the  research  and  development  cycle.* 

As  required  by  approved  test  plans,  test 
node  capabilities  will  include  the  ability  to 
duplicate,  simulate  or  otherwise  represent  end- 
to-end,  any  system  within  the  Naval  Telecom- 
munications System  and  will  be  representative 
of  a mobile  fleet  unit  and/or  systems  normally 
found  in  a Naval  Communications  Station. 

The  operation  and  maintenance  of  the 
Naval  Telecommunications  System  test  node 
are  the  responsibility  of  the  Navy.  Personnel 
utilization  for  agreed  upon  test  plans  will  be 


managed  by  the  Navy  in  coordination  with  the 
other  program  participants  in  suppo’t  of  the 
TRI-TAC  Joint  Test  and  Evaluation  Program. 

ADMINISTRATION 
AND  LOGISTICS 

The  TRI-TAC  Program  is  uniqie  in  that 
Service  resources  are  pooled  for  mutui.1  benefit 
to  develop  equipment  that  will  satisfy  common 
communications  requirements  in  the  tactical 
arena.  No  single  Service  could,  practically,  af- 
ford the  investment  requireu  ’o  develop  the 
various  TRI-TAC  sponsored  equipment.  A sig- 
nificant benefit  is  realizable  because  of  the  life- 
cycle  costs  savings  inherent  in  the  joint  support 
of  these  acquisitions  after  the  acquisitions  have 
been  fielded. 

The  responsibilities  of  the  TRI-TAC  Pro- 
gram participants  for  financial  management 
and  support  of  the  Joint  Test  Organization  are 
as  follows: 

• TRI-TAC  Office  - The  TRI-TAC  Office, 
in  collaboration  with  participating  DOD  com- 
ponents, will  determine  the  resources  required 
to  implement  the  operations  of  the  Joint  Test 
Organization.  The  office  must  assure  that  re- 
source requirements  are  reflected  adequately  in 
the  Five-Year-Defense  Program  and  budget 
documentation  prescribed  by  the  Office  of  the 
Secretary  of  Defense  for  various  events  in  the 
planning,  programming,  and  budgeting  system. 
In  addition,  the  office  must  maintain  adequate 
control  of  all  funds  made  available  to  support 
the  operations  of  the  joint  test  element  in  accor- 
dance with  current  policy  and  directives. 

• Military  Departments/DOD  Agencies  - 
The  Military  Departments  and  Agencies  will 
program  and  budget  for  resources  to  support 
the  joint  test  element  in  accordance  with  nor- 
mal procedures  prescribed  for  the  DOD  plan- 
ning, programming  and  budgeting  system. 
Funds  for  this  support  will  be  apportioned  on  a 
35,  35,  20,  10-percent  basis  among  the  Army, 
Air  Force,  Navy  and  the  National  Security 
Agency  respectively.  Joint  test  element  civilian 
personnel  costs  will  be  budgeted  by  the  Army 
and  will  be  included  as  part  of  the  apportioned 
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35  percent.’  This  funding  will  be  provided  to 
the  Director,  TRI-TAC  Office,  in  a timely 
manner,  consistent  with  the  availability  of 
funds. 

The  Military  Departments  and  Agencies 
will  program  and  budget  for  resources  for  the 
test  support  element  and  the  development  test 
and  evaluation  and  initial  operational  test  and 
evaluation  Service  test  officers  as  required.’ 
Funds  shall  be  retained  by  the  individual 
Department/ Agency  and  used  to  support  re- 
quirements of  the  pertinent  test  planning 
documents. 

The  Secretary  of  the  Army  will  provide 
logistic  and  administrative  support  to  all  ele- 
ments of  the  Joint  Test  Organization  with  reim- 
bursement as  appropriate.  Intra/inter-Service 
support  agreements  will  be  negotiated  by  the 
individual  Service/ Agency  with  the  joint  test 
facility  host.  Fort  Huachuca,  as  required.  In 
addition,  the  Secretary  of  the  Army  will  pro- 
gram funds  to  operate  and  maintain  the  facility 
support  element  of  the  Joint  Test  Organization. 

ADMINISTRATIVE  REQUIREMENTS 

Prior  to  the  start  of  development  test  and 
evaluation  and  initial  operational  test  and  eval- 
uation the  contractor,  under  the  auspices  of  the 
acquisition  Service  for  each  acquisition  item, 
will  install,  check  out  and  certify  the  equipment 
delivered  to  the  government  test  site  for  test. 
After  successful  completion  of  the  installation 
and  checkout,  the  acquisition  Service  represent- 
ative for  that  item  shall  provide  conditional 
technical  acceptance  to  the  contractor.  The 
acquisition  Service  representative  shall  insure 
that  all  delivery  items  are  complete,  to  include 
all  test  equipment,  other  supporting  equipment, 
spare  assemblies  and  parts,  technical  publica- 
tions to  include  maintenance  and  operator’s 
manuals,  programmer  guides,  calibration  pro- 
cedures, test  plans,  test  procedures,  data  sheets, 
contractor  development  test  results,  baseline 
drawings,  and  other  contractually  required 
items  identified  by  contractor  data  require- 
ments lists.  Transfer  of  property  control  from 
the  acquisition  Service  to  the  test  directors  for 
the  duration  of  development  test  and  evaluation 


and  initial  operational  test  and  evaluation,  will 
be  accomplished  by  utilizing  standard  military 
'forms/procedures. 

Upon  completion  of  development  test  and 
evaluation  and  initial  operational  test  and  eval- 
uation, test  documentation/assets  no  longer 
needed  at  the  test  sites  for  follow-on  testing  will 
be  returned  to  the  acquisition  Service.  Desig- 
nated TRI-TAC  sponsored  equipment  required 
to  form  the  baseline  for  other  tests  will  be 
transferred  from  the  acquisition  Services  to  the 
facility  support  element.  Property  control  of 
test  items  will  be  returned  to  the  acquisition 
Service  for  disposition  when  no  longer  required 
at  the  test  site. 

Physical  security  for  the  central  test  facil- 
ity and  the  logistics  support  test  facility  will  be 
provided  by  the  joint  test  element.  The  acquisi- 
tion Services  will  be  responsible  for  providing 
security  for  the  radio  relay  sites,  remote  operat- 
ing areas,  and  contractor  support  facility. 

Periodic  administrative  status  reports  pre- 
pared by  the  joint  test  element,  and  coordinated 
with  the  Joint  Test  Organization  will  be  submit- 
ted to  the  TRI-TAC  Office  and  all  test  and 
evaluation  participants. 

In  summary,  the  test  and  evaluation  for 
TRI-TAC  sponsored  equipment  must  be  jointly 
planned,  conducted  and  reported  by  the  Serv- 
ices and  Agencies  in  coordination  with  the 
TRI-TAC  Office.  Each  acquisition  must  be 
witnessed/observed  by  the  Services  and  Agen- 
cies during  contractor  testing  to  assure  that  the 
item  is  ready  for  government  testing.  Each 
tasked  acquisition  Service/ Agency  must  direct 
the  conduct  of  government  testing  in  a joint 
arena  and  insure  that  adequate  development 
test  and  evaluation  and  initial  operational  test 
and  evaluation  reports  are  prepared  so  that 
each  Service,  Agency,  the  TRI-TAC  Office,  the 
Deputy  Undersecretary  of  Defense  Research 
and  Engineering  (Test  and  Evaluation)  and  the 
Joint  Chiefs  of  Staff  can  prepare  independent 
evaluations  for  use  by  the  decisionmakers.  The 
TRI-TAC  Office  must  insure  that  DOD-level 
visibility  of  the  test  and  evaluation  program  is 
achieved  by  timely  submission  of  integrated  test 
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plans  for  each  acquisition.  Thus,  it  can  be  seen  tion,  coordination  and  support  of  all  TRI-TAC 
that  a successful  test  and  evaluation  program  is  Program  participants, 
dependent  upon  the  communication,  coopera- 
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HOW  MUCH  TESTING  IS  ENOUGH? 

by 

Howard  W.  Kreiner,  Systems  Planning  Corp. 


When  program  managers  and  Service  staffs  are  working  with  the  Offlce  of  the  Undersec- 
retary of  Defense  Research  and  Engineering,  a question  often  heard  is,  “How  much  testing  is 
enough”?  Test,  of  course,  covers  so  wide  a spectrum  of  activities  that  almost  everything  that  is 
not  actual  hardware  fabrication  could  be  somehow  included.  In  our  context  the  subject  is  the 
development  and  operational  testing  of  total  systems,  and/or  major  subsystems.  This  testing 
must  be  planned,  conducted,  and  evaluated  before  the  key  decisions  to  enter  full  scale  develop- 
ment, and  later,  production  of  the  system  can  be  done. 


AN  ANSWER 

The  answer  to  the  question,  “How  much 
testing  is  enough,”  must  take  into  account  the 
amount  of  testing  that  safely  can  be  deferred 
until 

• after  the  production  decision; 

• during  initial  production;  and, 

• after  field  deployment  of  the  system. 


The  simple  but  unsatisfying  answer  to  the 
question  is:  There  has  been  enough  testing 
when  you  can  make  a satisfactory  estimate  of 
the  risks  of  proceeding  into  full  scale  develop- 
ment, or  into  production,  and  are  able  to  bal- 
ance these  risks  against  the  effects  on  US  defen- 
ses of  program  cancellation.  The  answer  is 
unsatisfying.  First,  a budget  cannot  be  framed 
from  that  answer.  Second,  the  Initial  Opera- 
tional Capability  (IOC)  date  cannot  be  calcu- 
lated from  the  answer.  Moreover,  the  answer 
has  a disturbingly  subjective  tone.  The  implica- 
tion is  that  judgment  about  the  total  amount  of 
testing  is  not  made  until  test  results  are  in  hand, 
rather  than  at  the  start  of  the  program. 

The  nature  of  the  answer,  with  its  high 
degree  of  subjectivity,  makes  it  difficult  to  resist 
pressures  to  eliminate  or  minimize  the  required 
predecision  test  phase. 

84 


The  pressures  that  operate  against  ,com- 
prehensive  testing  are  obvious  and  measurable. 
The  longer  the  period  between  prototype  com- 
pletion and  production  initiation,  the  longer 
and  more  costly  the  R&D  phase,  including  test- 
ing, will  be.  Production  facilities  and  staff  re- 
quire funds  for  maintenance  even  if  production 
is  not  occurring.  The  inflation  meter  runs  while 
time  passes.  Unit  costs  rise  in  a manner  that 
currently  results  in  reduced  total  buys  of  equip- 
ment. And  most  evident,  equipment  is  delayed 
in  entering  service. 

In  favor  of  deferred  testing  with  its  appar- 
ent savings  of  time  and  money  it  is  noted  that 
service  introduction  leads  to  in-service  accumu- 
lation of  operating  hours  at  a faster  rate,  lower 
cost,  and  with  a realism  that  prototype  R&D 
testing  lacks.  Service  introduction  of  a system 
does  require  consideration  insofar  as  testing  is 
performed  to  obtain  intimate  familiarity  with 
the  operational  capabilities  of  a system. 

But  it  should  not  be  necessary  to  argue  the 
merits  of  a reasonable  amount  of  initial  opera- 
tional system  testing.  Against  all  the  pressures 
to  reduce  Operational  Test  and  Evaluation, 
there  is  the  balancing  risk  of  committing  the 
Department  of  Defense  to  the  purchase  of  unsa- 
tisfactory equipment.  The  reasons  for  testing 
have  been  affirmed  at  length  in  the  report  of  the 
Presidential  Blue  Ribbon  Defense  Panel  whose 
members  were  appointed  by  the  President  of 
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the  United  States.  These  reasons  have  been 
reinforced  at  intervals  by  reports  of  the  Defense 
Science  Board  Task  Force  on  Test  and  Evalua- 
tion (April  1974  and  February  1977).  What  is 
required  is  a clearer  definition  of  the  opera- 
tional meaning  of  words  like  “satisfactory,” 
“reasonable,”  and  “risk.” 

Of  these  terms,  the  most  difficult  word  to 
define  is  "risk.”  “Risk”  in  this  context  is  unmis- 
I takably  subjective,  for  the  risk  is  a subjective 

I judgment  as  to  whether  or  not  any  of  a myriad 

i of  unwelcome  outcomes  will  occur  in  conse- 

i quence  of  a decision  to  proceed.  As  stated 

i earlier,  the  counter  arguments  to  proceeding 

deliberately  tend  to  have  more  weight.  These 
[ counter  arguments  can  be  made  definite  and 

quantitative,  i.e.,  time  costs  money.  The  deci- 

> sion  to  proceed  takes  one  into  qualitative  judg- 

I ment  that  no  amount  of  cosmetic  language, 

such  as  “subjective  probabilities,”  can  turn  into 
quantitative  measurement.  There  are  some 
quantitative  aspects:  Engine  qualification  speci- 
fications, fatigue  test  practices,  reliability  de- 
sign methods  and  standards  all  are  risk  reduc- 
ing practices  that  enable  quantitative  measure- 
ment. But  the  task  of  risk  assessment  remains. 
Without  initial  operational  test  and  evaluation 
a method  does  not  exist  for  ascertaining  or  for 
measuring  in  advance  the  probability  that  a 
design  will  be  faulty  to  a severe  degree.  Exam- 
ple: How,  without  operational  test  could  you 
determine  that  a design  would  be  faulty  to  such 
an  extent  that  4400  trucks  would  have  to  go 
directly  from  production  line  to  repair  depot  for 
storage  to  await  a redesigned  transmission? 
Good  design  can  insure  that  parts  known  to 
need  regular  servicing  are  quickly  and  easily 
accessible.  Design  cannot  be  expected  to  take 
account  of  a part  breakdown  that  occurs  more 
frequently  in  the  operational  environment  than 
anticipated,  and  is  a part  that  requires  major 
dissassembly  to  reach. 

The  judgment  of  risk  and  test  adequacy 

> cannot  be  made  until  test  results  are  in  hand.  In 

I the  real  world  a manager  or  decision  maker 

would  not  allow  a judgment,  formed  by  match- 
[ ing  test  results  to  the  criteria  propos^  at  the 

I start  of  the  program,  to  stand  without  review 

1 when  test  results  become  available.  Circum- 

stances may  change  between  establishing  the 


criteria  and  getting  test  results  or,  the  implica- 
tions of  the  circumstances  may  be  differently 
perceived.  For  example,  to  a decision  maker  at  | 

program  initiation  the  prospect  of  a 30  percent  j 

cost  overrun  may  appear  unacceptable.  Cost 
overrun  may  prove  unavoidable,  unpleasant, 
but  nonetheless  acceptable  if,  after  the  program  j 

test  results  are  evaluated  another  course  of  j 

action  does  not  offer  a better  alternative.  To  a 
manager  faced  with  getting  a complex  piece  of 
equipment  built  and  working,  a possible  dura-  j 

bility  limitation  with  a propulsion  unit  may  ] 

seem  a manageable  problem  for  solution  by  a 
production  team.  After  the  euphoria  of  getting 
the  equipment  working  has  passed,  a public  1 

debate  on  the  durability  problem  might  be  j 

something  the  same  program  manager  would  ] 

prefer  to  avoid  until  after  a further  serious  j 

effort  is  made  to  solve  the  “manageable”  prob- 
lem and  to  confirm  the  solution.  i 

ACHIEVEMENT-PACED  SCHEDULING  ! 

The  after-the-fact  character  of  decisions 
based  on  test  results  translates  into  a well-  j 

known,  but  infrequently  implemented  policy  of:  | 

Achievement-paced  scheduling.  The  difficulties  ■ 

of  this  policy  include  ail  the  reasons  described  1 

earlier  for  deferring  test,  and  the  well-known 
problems  of  making  an  achievement-paced  pro- 
gram fit  the  constraints  of  a yearly  budget 
cycle.  The  difficulties  are  nowhere  more  evident 
than  when  achievement-paced  scheduling  is  ap- 
plied to  system  reliability. 

A method  has  not  been  found  to  eliminate 
the  difficulties,  but  there  is  one  technique  that 
can  mitigate  them.  The  method  is  to  develop 
and  organize  experience  into  meaningful  check- 
lists of  essential  test  actions.  Comprehensive 
check  lists  were  published  and  distributed 
widely  by  the  DD(T&E)  in  conjunction  with 
the  report  of  the  Defense  Science  Board  Task 
Force  on  Test  and  Evaluation  in  April  1974. 

The  series  of  checklists  was  present^  in  nine 
volumes,  in  addition  to  the  general  checklists 
and  software  test  proposals  contained  in  the 
Task  Force  report.  The  nine  volumes  addressed 
individual  generic  system  types:  aircraft,  mis- 
siles, ships,  ground  vehicles,  antisubmarine 
warfare  systems,  airborne  electronic  counter- 
measures systems,  airborne  general  surveillance 


Vol  I,  No.  s 


85 


radar  systems,  command  and  control  systems, 
and  common  test  gear.  These  lists  remain  appli- 
cable and  can  assist  in  framing  and  putting  a 
workable  scale  on  the  answer  to  the  question: 
“How  much  testing  is  enough”? 


It  is  essential  that  the  fundamentally  sub- 
jective, indefinite  answer  be  remembered:  There 
is  enough  testing  when  it  is  possible  to  make  a 


satisfactory  estimate  of  the  risks  of  proceeding 
into  full  scale  development,  or  into  production, 
to  balance  against  the  effects  on  US  defense  of 
cancelling  the  program.  This  concept  needs 
adequate  weight  in  the  program  development 
process,  with  special  attention  devoted  to  the 
identification  of  essential  test  actions.  Without 
successful  concept  accomplishment  and  identi- 
fication of  essential  test  actions  one  should  not 
proceed  to  the  next  major  phase  of  the  program. 


Mr.  Howard  W. 

Kreiner  is  a Staff  Member 
with  System  Planning  Corp. 

From  1972  until  mid-1977 
he  was  a Staff  Member  in  the 
office  then  designated  as  Of- 
fice of  the  Deputy  Director, 

Defense  Research  and  Engi- 
neering (Test  and  Evaluation).  As  a Staff  Member,  Mr. 
Kreiner  served  as  Executive  Secretary,  Defense  Science 
Board  Task  Force  on  Test  and  Evaluation  (1973-1974). 
He  again  held  this  position  in  1976  when  the  Task  Force 
was  reconstituted  for  a renewed  look  at  Test  and  Evalua- 
tion policy. 

Since  1947,  Mr.  Kreiner  has  served  as  an  Operations 
Research  Analyst.  He  worked  with  the  Navy  Operations 
Evaluatioh  Group  and  its  successor,  the  Center  for  Naval 
Analyses,  until  December  1966.  Assignments  included  a 
l-year  tenure  with  the  Air  Test  and  Evaluation  Squadron 
1 (VX-1)  plus  assignments  at:  Headquarters,  Office  of  the 
Commander,  Operational  Test  and  Evaluation  Force  and. 
Office  of  the  Commander  in  Chief,  Atlantic  Fleet.  In  addi- 
tion, Mr.  Kreiner  has  served  under  the  Commanders  of 
the  Sixth  and  Seventh  Fleets. 

Mr.  Kreiner  is  a founding  member  of  the  Operations 
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Chemical  Engineering  from  the  University  of  Pennsylva- 
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TEST  AND  EVALUATION  POLICY 

by 

Charles  W.  Karns,  Staff  Assistant 
Office  of  the  Undersecretary  of  Defense  Research  & Engineering 

The  large  sums  spent  each  year  hy  the  Department  of  Defense  to  maintain  an  adequate 
military  defense  of  the  United  States  require  valid  decisions  relative  to  the  systems  proposed 
for  acquisition.  Those  decisions  in  the  system  acquisition  process  must  he  based  on  valid  test 
and  evaluation  (T&E).  Two  primary  objectives  of  current  I>OD  T&E  programs  are  to  measure, 
accurately,  the  operational  characteristics  of  new  weapon  systems  and  to  make  a realistic 
evaluation  of  operational  effectiveness  and  suitability  of  these  systems. 


INTRODUCTION 

The  estimated  $2.7  billion  that  the  De- 
partment of  Defense  (DOD)  will  spend  on  test 
and  evaluation  activities  in  FY  1978,  while  only 
a fraction  of  total  defense  expenditures,  is  a 
large  sum. 

There  are  two  principal  types  of  test  and 
evaluation  conducted  on  new  military  systems: 

• Development  Test  and  Evaluation 
(DT&E).  Development  Test  and  Eval- 
uation is  conducted  by,  or  under  the 
supervision  of,  a development  agency 
to  evaluate  the  technical  performance 
of  prototype  equipment.  This  testing  is 
generally  conducted  by  skilled  techni- 
cians and  engineers  under  controlled 
conditions. 

• Operational  Test  and  Evaluation 
(OT&E).  Operational  Test  and  Evalu- 
ation is  conducted  by  military  person- 
nel to  determine  the  degree  to  which 
new  equipment  fulfills  military  opera- 
tional requirements.  This  type  of  test- 
ing is  conducted  under  conditions  that 
closely  duplicate  the  environment  ex- 
pected in  Held  operations.  Operational 
testing  includes  maintenance  and  logis- 
tics support,  operation  of  the  equip- 
ment, and,  where  appropriate,  the  ev^- 
uation  of  tactics  and  techniques  for  the 


most  effective  system  use  in  combat. 
Operational  tests  are  conducted  on  re- 
search and  development  prototypes  of 
new  equipment  as  well  as  on  early  pro- 
duction models. 

The  DOD  officials  who  make  decisions 
regarding  the  acquisition  of  new  systems  de- 
pend upon  results  of  test  and  evaluation.  Specif- 
ically, test  and  evaluation  provides  information 
on 

• how  well  a program  is  meeting  current 
objectives  and, 

• what  the  ultimate  outcome  of  the  pro- 
gram is  likely  to  be. 

Generally,  Development  Test  and  Evaluation 
most  often  provides  information  of  the  first 
kind  by  indicating  whether  a system  under 
development  is  meeting  the  technical  specifica- 
tions. Operational  Test  and  Evaluation  usually 
provides  information  about  how  well  a system 
will  perform  once  it  becomes  operational. 

Although  decisionmakers  require  other 
types  of  information,  the  information  provided 
by  DT&E  and  OT&E  is  essential  to  most  major 
decisions  made  during  the  course  of  a program. 
Since  testing  is  expensive,  a reasoned  judgment 
must  be  made  as  to  how  much  testing  is  needed. 


or  how  much  risk  can  be  accepted  in  making 
program  decisions.  The  factors  that  influence 
the  judgment  about  how  much  testing  is  needed 
vary  from  program  to  program.  Hence,  the 
amount  of  test  and  evaluation  to  be  accom- 
plished may  also  vary. 

Aside  from  being  essential  for  decision- 
making, test  and  evaluation  is  important  in 
other  ways.  Development  testing,  for  example, 
is  an  integral  part  of  the  system  design  and 
development  process.  In  this  process  a system 
design  is  converted  into  hardware.  The  hard- 
ware is  tested,  deficiencies  are  noted,  and  the 
design  is  modified  as  necessary.  This  process  is 
iterated  until  the  system  hardware  reaches  a 
final  design  configuration.  Development  Test 
and  Evaluation  provides  the  system  designer 
with  feedback  jnformation  that  would  be  diffi- 
cult or  impossible  to  obtain  by  analytical 
processes. 

Operational  testing  has  a direct  relation  to 
the  military  capabilities  of  operating  forces. 
The  more  effort  that  is  made  to  identify  and 
correct  operational  deficiencies  before  the  de- 
sign of  a system  is  frozen,  the  greater  the 
military  capability  of  the  operating  forces  when 
the  new  weapons  system  is  deployed.  Also 
significant  cost  benefits  may  be  realized  when 
major  system  deficiencies  are  corrected  prior  to 
operational  deployment.  Detection  and  correc- 
tion of  deficiencies  prior  to  deployment  is  usu- 
ally less  expensive  and  less  time-consuming 
than  is  retrofitting  systems  after  the  system  has 
entered  the  operational  inventory.  Also,  such 
detection  and  correction  is  less  disruptive  to 
essential  activities  engaged  in  during  deploy- 
ment, such  as  maintaining  a state  of  operational 
readiness,  conducting  training,  and  developing 
tactics. 

DOD  TEST  AND  EVALUATION 
ORGANIZATION 

In  1970  the  Presidential  Blue  Ribbon  De- 
fense Panel  reported  that  operational  test  and 
evaluation  in  the  Department  of  Defense  had 
been  infrequent,  poorly  designed  and  executed, 
and  generally  inadequate.  In  response,  test  and 
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evaluation  activities  were  restructured — grea- 
ter emphasis  was  placed  on  the  role  of  test  and 
evaluation  in  the  weapons  system  acquisition 
process.  The  recommendations  of  this  Panel 
were  subsequently  strengthened  by  the  Com- 
mission on  Government  Procurement. 

In  response  to  the  recommendations  of 
these  groups,  the  DOD; 

• Created  a higher-than-Service  organi- 
zation, specifically  the  Office,  Deputy 
Director  of  Defense  Research  and  En- 
gineering (Test  and  Evaluation) 
DD(T&E),  to  oversee  Defense  test  and 
evaluation  activities. 

• Established  within  the  Office, 
DD(T&E)  a capability  to  initiate  and 
conduct  operational  tests  and  evalua- 
tion in  joint  operations,  that  is,  test  and 
evaluation  involving  more  than  one 
Service. 

• Strengthened  the  Services’  operational 
testing  of  newly  developed  systems  and 
required  that  the  performance  of  the 
systems  be  tested  (in  an  environment 
approximating  the  expected  opera- 
tional conditions),  and  evaluated  prior 
to  commitment  of  the  system  to  full 
production. 

Within  each  of  the  Military  Departments 
there  has  been  established  an  operational  test 
agency,  independent  of  both  the  developer  and 
user.  The  operational  test  agencies  are  charged 
with  conducting  operational  evaluations  of  new 
equipment. 

DSARC  PROCESS 

The  current  approach  to  managing  weap- 
ons systems  acquisition  in  DOD  is  based  on  the 
milestone  concept.  At  each  significant  mile- 
stone in  the  life  of  a major  weapon  acquisition 
program  its  progress  is  carefully  reviewed  to 
determine  if  the  program  should  be  advanced  to 
its  next  phase.  These  reviews  are  conducted  at 
four  major  program  milestones. 
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Milestone  0 marks  the  point  at  which 
exploration  of  alternative  solutions  to  a defined 
mission  need  is  begun  and  the  program  is 
initiated. 

Milestone  / is  the  point  where  one  or  more 
system  alternatives  are  selected  for  demonstra- 
tion and  validation  of  ability  to  fulfill  the  de- 
fined need. 

Milestone  II  is  where  a decision  is  made 
on  whether  to  enter  full-scale  development  of 
the  system. 

Milestone  ///is  the  final  milestone.  Here 
a decision  is  made  on  whether  to  initiate  full- 
scale  production  and  deployment  of  the  system. 

In  addition  to  these  fixed  milestones,  pro- 
gram reviews  may  be  held  at  any  point  in  the 
life  of  a program  when  major  issues  arise  or 
when  the  program  breaches  or  threatens  to 
breach  a prescribed  cost,  schedule,  or  perfor- 
mance threshold.  Most  information  used  at 
reviews  to  assess  program  progress  and  likely 
success  is  derived  from  test  and  evaluation 
results.  Depending  on  the  assessments  made  at 
the  reviews,  a program  may  be  advanced  to  its 
next  phase,  cancelled,  reoriented,  or  held  in  its 
present  phase. 

Major  program  reviews  conducted  at  the 
OSD  level  are  the  responsibility  of  the  Defense 
Systems  Acquisition  Review  Council 
(DSARC).  The  DSARC  reports  its  recommen- 
dations directly  to  the  Secretary  of  Defense, 
who  makes  the  final  decision  on  whether  the 
program  is  to  be  advanced  to  its  next  phase. 
The  principal  members  of  the  DSARC  have 
been  OSD  officials  at  the  Assistant  Secretary 
level — the  Director  of  Defense  Research  and 
Engineering,  the  Comptroller,  the  Assistant 
Secretary  of  Installations  and  Logistics,  and  the 
Director  of  Planning  and  Evaluation.  Although 
some  changes  in  the  precise  membership  of  the 
DSARC  will  follow  the  recent  OSD  reorganiza- 
tion, membership  will  consist  of  those  officials 
responsible  for  the  functional  areas  represented 
in  the  past.  The  Deputy  Director  of  Research 
and  Engineering  (Test  and  Evaluation),  not  a 
DSARC  principal,  normally  participates  in  the 


Council’s  deliberations  and  provides  to  its 
members  an  assessment  of  the  testing  already 
completed  and  the  adequacy  of  the  plans  for 
tests  to  be  conducted  in  the  f^uture.  Thereafter, 
the  DD(T&E)  submits  a written  assessment  di- 
rectly to  the  Secretary  of  Defense  for  use  in  re- 
viewing DSARC  recommendations. 

A new  acquisition  program  may  be  initia- 
ted (at  Milestone  0)  when  a DOD  Component 
Head  perceives  the  existence  of  a mission  need 
and,  having  submitted  a statement  of  that  mis- 
sion need  to  the  Secretary  of  Defense,  requests 
approval  to  identify  and  explore  alternative 
solutions. 

A Decision  Coordinating  Paper,  prepared 
for  use  at  the  time  of  Demonstration  and  Vali- 
dation Decision  (Milestone  I),  identifies  critical 
issues  and  areas  of  risk  to  be  addressed  by  test 
and  evaluation.  The  Decision  Coordinating  Pa- 
per provides  the  test  objectives  and  measures  of 
effectiveness  related  to  satisfaction  of  mission 
need,  and  a summary  of  schedules  and  resource 
requirements  applicable  to  test  activity  prior  to 
Milestone  II.  The  DSARC  (or  the  Systems 
Acquisition  Review  Council  (SARC)  in  one  of 
the  Services  for  those  programs  delegated  to  it) 
determines  the  adequacy  of  the  critical  issues, 
test  objectives,  and  test  schedules. 

When  full-scale  engineering  development 
is  proposed  (Milestone  II),  the  revised  Decision 
Coordinating  Paper  gives; 

• the  results  of  test  and  evaluation  ac- 
complished to  date; 

• an  updated  statement  of  critical  issues, 
test  objectives,  and  areas  of  risk  need- 
ing further  assessment; 

• a summary  of  performance  criteria 
goals  and  thresholds; 

• an  ov  erview  of  test  plans,  milestones, 
and  program  interrelation. 

Supporting  details  of  test  plans  and  test  results 
are  made  available  as  requested  by  the  Deputy 
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Director  (T&E).  The  DSARC  (or  designated 
Service  Council)  assesses  and  comments  to  the 
Secretary  of  Defense  as  to  the  adequacy  of  test 
and  evaluation  progress,  and  of  the  test  and 
evaluation  planned  to  occur  prior  to  the  first 
major  production  decision. 

At  Milestone  III  results  of  testing  and 
plans  for  future  testing  are  again  assessed,  and 
the  DSARC  (or  designated  Service  Council) 
comments  to  the  Secretary  of  Defense  as  to  the 
adequacy  of  these  test  results  to  support  a 
decision  relative  to  full  scale  system  production 
and  deployment. 

Because  test  and  evaluation  is  essential  to 
the  decisionmaking  process,  it  is  important  that 
the  Deputy  Director  (T&E)  be  independent  and 
objective  in  making  assessments.  Current  DOD 
directives  provide  the  means  by  which  such 
independence  and  objectivity  can  be  exercised. 
Example;  Even  though  the  Director  of  Defense 
Research  and  Engineering  is  the  direct  superior 
of  the  Deputy  Director  (T&E),  the  latter  has 
direct  access  to  the  Secretary  of  Defense,  partic- 
ularly when  weapons  system  acquisition  deci- 
sions are  to  be  made.  In  this  respect,  the  posi- 
tion of  the  Deputy  Director  (T&E)  is  very 
similar  to  that  of  the  heads  of  the  independent 
test  agencies  within  the  Services. 

NOTE:  Involvement  of  the  Deputy  Di- 
rector (T&E)  in  a program  is  not  confmeu  to 
the  limited  period  of  time  required  to  p epare 
for  the  conduct  DSARC  reviews.  If  the 
DD(T&E)  were  to  wait  until  the  meeting  of  the 
DSARC  to  report  that  the  testing  of  a system 
was  unsatisfactory,  unnecessary  additional  pro- 
gram costs  could  accrue.  A loss  of  valuable  time 
could  result  because  of  the  need  to  develop  new 
test  models,  or  modify  existing  ones,  and  the 
requirement  to  repeat  tests.  It  is  most  effective 
for  the  DD(T&E)  staff  to  be  involved  in  tests  on 
a daily,  weekly  and  monthly  basis — from  the 
test  planning  through  the  final  evaluation  of 
test  results.  The  action  officers  in  the  office  of 
the  Deputy  Director  (T&E)  bring  to  bear  a 
military  operational  background  and  experi- 
ence gained  from  observing  large  numbers  of 
tests  on  a wide  variety  of  equipments,  as  well  as 
an  objectivity  nurtured  by  remoteness  from  the 


development  process.  By  reviewing  test  plans 
before  the  tests  are  initiated  and  by  observing 
results  as  the  test  proceed  and  even  actively 
participating  in  certain  tests,  the  Deputy  Direc- 
tor (T&E)  can  advise  a Program  Manager  early 
in  the  development  cycle  as  to  whether  or  not 
the  equipment  is  advancing  toward  meeting  the 
standards  by  which  the  weapon  system  will  be 
judged.  Thereby  time  is  allowed  for  corrective 
action  to  be  taken  prior  to  the  milestone  deci- 
sion point. 

RECENT  DEVELOPMENTS 

The  office  of  the  Deputy  Director  (T&E) 
is  constantly  trying  to  improve  the  quality  of 
test  and  evaluation  information  available  for 
acquisition  decisionmaking.  Changes  are  being 
made  in  test  and  evaluation  policies  and  direc- 
tives to  emphasize  the  need  for  operational 
testing  early  in  the  weapon  system  development 
cycle.  In  the  early  1970’s  emphasis  was  placed 
almost  exclusively  on  conducting  Operational 
Test  and  Evaluation  just  prior  to  the  Milestone 
III  major  production  decision.  This  was  be- 
cause of  the  large  procurement  funding  com- 
mitment made  at  Milestone  III.  Lately,  it  has 
been  realized  that  earlier  Operational  Test  and 
Evaluation  may  be  needed  for  four  reasons: 

• In  considering  system  concepts  during 
the  early  stages  of  a program,  it  is  nec- 
essary to  know  if  the  concepts  are  tech- 
nically feasible,  and  operationally 
viable,  and  tactically  sound. 

• To  have  impact  on  the  research  and  de- 
velopment phase  of  a program.  Opera- 
tional Test  and  Evaluation  must  be 
conducted  much  earlier  than  at 
DSARC  III.  By  DSARC  III  most  re- 
search and  development  resources  will 
have  been  expended. 

• The  earlier  in  the  development  cycle  a 
deficiency  is  identified  and  corrected, 
the  less  the  cost  of  corrective  action. 

• A trend  toward  initiating  limited  pro- 
duction before  the  major  production 
decision  results  in  an  earlier  commit- 
ment of  procurement  funds. 
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For  these  reasons,  emphasis  now  is  on 
conduct  of  Operational  Test  and  Evaluation  as 
early  as  possible  in  the  acquisition  cycle.  Early 
OT&E  insures  that  relevant  test  information  is 
available  for  decisionmaking  during  the  early 
development  stages  of  a program. 

A particular  area  in  which  test  and  evalua- 
tion is  making  a significant  contribution  is  that 
of  equipment  reliability.  This  is  not  a new  area 
of  interest.  The  Services  have  long  concentrated 
on  reliability  and  have  given  close  attention  to 
the  achievement  of  improved  reliability 
through  proper  design  and  through  quality  con- 
trol in  the  production  process.  The  Oflice,  Dep- 
uty Director  (T&E)  is  in  full  agreement  with 
these  efforts,  and  works  cooperatively  with  the 
Service  development  and  materiel  agencies  that 
carry  out  design  and  quality  control  actions. 
Test  and  Evaluation  plays  a major  role  in  the 
measurement  of  reliability  and  the  isolation  of 
failures.  The  actual  correction  of  defects  (for 
the  most  part)  is  accomplished  by  redesign  of 
the  equipment  or  improved  quality  control  in 
equipment  manufacture. 

Laboratory  and  bench  tests  of  compo- 
nents and  assemblies  by  the  contractor  are  the 
mainstay  of  reliability  testing  and  form  the 
basis  for  proving  that  equipment  complies  with 
contract  specifications.  Often  such  tests  are 
required  to  further  pinpoint  and  analyze  field 
failures  so  that  engineering  corrections  can  be 
devised. 

What  laboratory  and  bench  tests  do  not  do 
is  expose  the  system  to: 

• the  full  range  of  operational  stresses; 

• the  harder  handling  of  less  well-trained 
military  operation  and  maintenance 
personnel; 

• the  cumulative  effect  of  harsh  field 
conditions. 

If  reliability  weaknesses  are  to  be  discovered 
before  large  numbers  of  equipment  are  de- 
ployed, the  equipment  must  be  tested  under 
typical  operational  conditions.  Therefore,  the 


Services  now  are  required  to  specify  reliability 
requirements  in  operational  terms.  These  re- 
quirements are  to  be  measured  under  opera- 
tional conditions  by  operational  personnel. 

A limiting  factor  in  operational  testing  is 
that  of  obtaining  sufficient  hours  of  system 
operation  under  the  very  conditions  that  make 
those  hours  most  costly;  that  is,  full  system 
operation  in  an  actual  field  environment.  To 
augment  these  valuable  hours  it  is  necessary  to 
insure,  to  the  extent  possible,  that  development 
testing  of  new  systems  is  done  under  realistic 
conditions.  The  failure  data  then  can  be  com- 
bined with  the  results  of  operational  testing,  to 
give  a comprehensive,  yet  the  most  economical 
method  of  evaluating  weapon  system  reliability. 

Closely  allied  with  reliability  is  production 
quality  control  and  quality  assurance.  The  esti- 
mate of  reliability  that  can  be  achieved,  based 
on  prototype  testing,  will  not  be  realized  with- 
out adequate  production  line  procedures  and 
tests.  Although  production  engineering  and 
quality  control  are  not  functions  of  the  Office  of 
the  Deputy  Director  (T&E),  close  coordination 
must  be  exercised  with  those  who  are  charged 
with  performing  those  functions.  The  early 
equipment  production  models  will  be  subjected 
to  follow-on  OT&E,  that  is,  operational  test  of 
production  models  in  a tactical  environment. 
Both  those  responsible  for  test  and  evaluation 
and  those  responsible  for  production  engineer- 
ing expect  the  production  models  to  exhibit 
maturity. 

Reliability  is  just  one  of  the  measures  of 
system  performance  for  which  concern  must  be 
expressed.  Regardless  of  how  performance  is 
measured,  a continuing  goal  must  be  early  de- 
tection of  major  system  deficiencies  if  such  defi- 
ciencies exist.  Early  detection  and  correction  of 
deficiencies  can  result  in  more  operating  force 
capability  and  less  likelihood  of  costly  retrofit 
programs. 

Although  test  and  evaluation  is  making  a 
significant  contribution  to  the  weapons  systems 
acquisition  process,  it  is  the  job  of  the  Deputy 
Director  (T&E)  to  insure  that  unnecessary  tests 
are  not  conducted.  There  is  a point  at  which  the 


incremental  value  of  information  obtained  by 
additional  testing  is  not  worth  the  expense.  To 
the  extent  possible,  each  test  conducted  must 
contribute  to  the  knowledge  of  the  essential 
capability  of  a system  in  rough  proportion  to 
the  resources  committed  to  the  test. 

In  1976  the  Director  of  Defense  Research 
and  Engineering  asked  the  Defense  Science 
Board  to  form  a Task  Force,  under  the  Chair- 
manship of  Dr.  Eugene  Fubini,  to  look  into  the 
questions  of  emphasis  placed  on  test  and  evalu- 
ation and  the  amount  of  time  and  money  de- 
voted to  these  activities.  This  Task  Force  af- 
firmed the  fundamental  and  integral  character 
of  test  and  evaluation  in  the  system  acquisition 
process.  The  major  test  and  evaluation  system 
elements,  that  is,  the  DSARC  and  its  interfaces 
with  the  Office  of  the  Deputy  Director  (T&E), 
were  found  to  function  well. 

In  the  view  of  the  Task  Force,  under 
applicable  DOD  Directives,  there  appears  to  be 
little  or  no  overtesting  done.  The  testing  done 
contributes  its  full  value  to  the  improvement 
and  verification  of  system  performance.  It  was 
concluded  that  the  management  overview  pro- 
vided by  the  Office  of  the  Deputy  Director 
(T&E)  adds  to  the  completeness  and  quality  of 
system  testing.  The  Task  Force  believes  that  the 
Deputy  Director  (T&E)  does  in  fact  apply  the 
precepts  recommended  by  an  earlier  Defense 
Science  Board  Task  Force  (1974). 

The  main  concern  of  the  Task  Force  is 
that  within  DOD  a mechanism  does  not  exist  to 
insure  that  risk  assessments  are  made  on  pro- 
grams wherein  the  test  results  are  not  alto- 
gether satisfactory  and  correction  of  deficien- 
cies is  indicated.  The  Task  Force  recognized 
that  such  risk  assessments  go  beyond  the  re- 
sponsibilities of  the  Deputy  Director  (T&E). 

The  Task  Force  was  concerned  that  rigid 
application  of  test  and  evaluation  policy  direc- 
tives sometimes  leads  to  the  so-called  “T&E 
Gap.”*  The  report  restated  the  alternatives  for 


•A  break  in  testing  between  the  end  of  the  development 
pha.se  and  the  beginning  of  production  (caused  by  the 
requirement  for  completion  of  adequate  testing  prior  to  a 
production  decision). 
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avoiding  the  gap  proposed  by  the  earlier  De- 
fense Science  Board  Task  Force  on  Test  and 
Evaluation.  These  alternatives  are 

• early  provisions  for  additional  research 
and  development  hardware,  to  be  re- 
search and  development  funded  and 
built  for  Initial  Operational  Test  and 
Evaluation. 

• an  additional  phase  of  testing  to  cover 
the  T&E  Gap; 

• advanced  availability  of  production 
items  through  early  availability  of  long 
lead  time  production  funding  and  low 
rate  initial  or  pilot  production  funding. 

Response  to  the  Task  Force’s  recommen- 
dations is  contained  in  a forthcoming  re- 
issuance of  DOD  Directive  5000.3,  “Test  and 
Evaluation.”  The  DOD  Directive  5000.3  is  re- 
sponsive also  to  revised  DOD  Directive  5(XX).  1, 
“Major  System  Acquisition,”  and  DOD  Direc- 
tive 5000.2,  “Major  System  Acquisition  Proc- 
ess.” Significant  changes  in  these  acquisition 
policy  directives  that  impact  on  test  and  evalua- 
tion are  the  establishment  of  the  Milestone  0 
Program  Initiation  Decision  and  the  creation  of 
a Systems  Acquisition  Review  Council  (SARC) 
in  each  of  the  Military  Departments.  These 
councils,  referred  to  as  Service  SARC,  advise 
the  respective  Secretaries  of  the  Military  De- 
partments in  a manner  equivalent  to  the 
DSARC  advising  the  Secretary  of  Defense. 

To  keep  acquisition  programs  in  close 
consonance  with  mission  needs,*  the  Secretary 
of  Defense  now  requires  submission  of  a Mis- 
sion Element  Need  Statement  (MENS)  when  a 
mission  need  is  perceived  to  exist  and  a new 
capability  is  required  to  meet  that  need.  Mile- 
stone 0 is  mark^  by  the  approval  of  the  MENS 
and  the  direction  to  one  or  more  DOD  compo- 
nents to  systematically  and  progressively  ex- 
plore and  develop  alternative  system  concepts 
to  satisfy  the  approved  need.  After  this,  when 
appropriate  and  prior  to  Milestone  I,  test  and 


•Required  by  OMB  Circular  A-1 09. 
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evaluation  is  accomplished  to  assist  in  selecting 
preferred  alternative  system  concepts  and  to 
assess  the  operational  impact  of  the  candidate 
technical  approaches. 

Under  the  procedures  set  by  revised  Di- 
rectives 5000.1  and  5000.2,  the  DSARC  will 
review  those  programs  classed  as  strategic,  nu- 
clear, joint-Service,  multinational,  intelligence, 
or  communications,  command,  and  control,  at 
Milestone  I.  The  DSARC  will  review  all  major 
programs  at  Milestones  II  and  III.  In  cases 
where  a DSARC  review  is  not  held,  and  in  most 
cases  where  a DSARC  is  held,  the  cognizant 
Service  SARC  is  to  advise  the  Service  Secretary 
relative  to  the  Secretary’s  recommendations  to 
the  DSARC  or  to  the  Defense  Acquisition 
Executive  and  the  Secretary  of  Defense.  Test 
and  evaluation  inputs  to  the  Service  SARC  are 
to  be  made  by  development  and  operational  test 
and  evaluation  agencies,  in  much  the  same 
manner  as  the  Deputy  Director  (T&E)  supports 
the  DSARC. 

Another  significant  positive  effect  on  test 
and  evaluation  arising  from  the  revised  DOD 
directives  on  acquisition  is  the  provision  for  the 
Secretary  of  Defense,  when  he  approves  the 
selection  of  a system  for  full-scale  engineering 
development  at  Milestone  II,  to  authorize 
procurement  of  long  lead  production  items  and 
limited  production  of  items  for  operational  test 
and  evaluation.  This  is  consistent  with  the 
recommendations  of  the  Defense  Science  Board 
Task  Force. 


I 
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In  summary,  recent  changes  in  test  and 
evaluation  policy  have  been  designed  to  provide 
effective  inputs  to  decisions  concerning  acquisi- 
tion of  major  systems.  The  purpose  of  test  and 
evaluation  is  to  evaluate,  under  conditions  as 


realistic  as  possible,  the  operational  effective- 
ness and  operational  suitability  of  new  systems. 
Possible  serious  deficiencies  are  to  be  high- 
lighted at  early  stages  of  development  to  avoid 
costly  remedies,  otherwise  required  in  later 
stages.  The  Deputy  Director  (T&E)  and  the 
entire  test  and  evaluation  community  are  con- 
tinuing efforts  to  provide  valid  inputs  to  the 
DOD  decisionmakers — decisionmakers  who 
must  make  valid  decisions  concerning  the  ac- 
quisition of  many  expensive  systems  for  the 
defense  of  the  country. 
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AIRCRAFT  OPERATIONAL 

TEST  AND  EVALUATION 

by 

Lt  Col  Michael  J.  Butchko,  Dept  of  the  Air  Force 

In  this  article  the  author  presents  the  evolution  of  operational  test  and  evaluation  (OT&E) 
within  the  Department  of  Defense  (DOD)  and  the  Air  Force.  Recommendations  are  provided 
for  the  Program  Manager  that  should  ensure  that  valid  operational  test  and  evaluation  require- 
ments are  in  the  development  programs.  Emphasis  is  placed  on  initial  operational  test  and 
evaluation  (lOT&E) — that  portion  of  operational  test  and  evaluation  accomplished  prior  to  the 
initial  production  decision — and  the  role  of  the  Air  Force  Test  and  Evaluation  Center 
(AFTEO. 


BACKGROUND 


Flight  test  of  powered,  heavier-than-air 
aircraft  has  been  going  on  in  the  United  States 
since  17  December  1903,  when  the  Wright 
Brothers  flew  what  was  probably  the  shortest 
successful  test  flight  on  record.  In  February, 
1977,  the  Space  Shuttle,  piggybacked  on  a Boe- 
ing 747  made  its  first  test  flight.  Both  events 
were  unique — both  events  were  exciting.  The 
first  event  represents  the  efforts  of  two  extraor- 
dinary men.  The  latter  event  represents  efforts 
of  thousands  of  persons.  Between  these  two 
aviation  milestones,  hundreds  of  aircraft  have 
gone  through  extensive  flight  test  programs 
with  varying  degrees  of  success.  While  test 
program  success  does  not  necessarily  result  in 
the  production  of  a new  aircraft,  an  unsuc- 
cessful test  program  just  about  guarantees  that 
a plane  will  not  be  produced. 


system.  However,  as  the  time  between  Mile- 
stone 0 (Program  Initiation)  and  Initial  Opera- 
tional Capability  (IOC)  grows  longer  and  more 
expensive,  the  Air  Force  is  accused  of  testing  to 
obsolescence.  The  pendulum  swings  back  and 
forth.  The  Program  Manager,  the  man  charged 
with  the  development  of  the  system,  has  the 
responsibility  to  ensure  that  the  program  has  an 
efficient  and  effective  test  program. 

Here  the  author  investigates  the  evolution 
of  operational  test  and  evaluation  (OT&E)  pol- 
icy in  the  Air  Force  aircraft  acquisition  process. 
The  methodology  employed  traces  Department 
of  Defense  and  Air  Force  policy  on  test  and 
evaluation  since  1970  with  particular  attention 
to  the  proposed  revision  of  DOD  Directive 
5000.3. 


The  test  program  has  accomplished  its 
purpose  if  it  verifies  that  the  system  under 
development  is  ready  for  production.  Also,  the 
test  program  has  accomplished  its  purpose  if  it 
identifies  a major  fault  that  leads  to  program 
cancellation.  If  significant  problems  surface  af- 
ter an  aircraft  is  in  the  operational  units,  the  Air 
Force  is  accused  of  fielding  an  underdeveloped 


DEFINITIONS 


To  preclude  confusion.  Development  Test 
and  Evaluation  (DT&E)  and  Operational  Test 
and  Evaluation  (OT&E)  are  defined  as  per 
DRAFT  DOD  Directive  5000.3  dtd  12  July 
1977. 
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• “DT&E  is  that  test  and  evaluation 
conducted  to  assist  the  engineering  de- 
sign and  development  process  and 
verify  attainment  of  technical  perfor- 
mance specification  and  objectives. 
DT&E  is  normally  managed  and  moni- 
tored by  the  DOD  components’  devel- 
oping agency....” 

• “OT&E  is  that  test  and  evaluation 
conducted  to  estimate  a system’s  oper- 
ational effectiveness  and  operational 
suitability....” 

OT&E  POLICY  1970-1976 

The  policies  governing  test  and  evaluation 
have  changed  over  the  years  in  response  to 
changing  needs  and  in  response  to  a variety  of 
internal  and  external  criticisms.  Since  1970,  the 
majority  of  policy  changes  and  criticisms  have 
been  directed  toward  the  operational  aspects  of 
testing. 

In  1970  the  flight  test  portion  of  the  air- 
craft acquisition  process  was  separated  into 
three  categories.  Category  I (CAT  I),  Subsys- 
tem Development  Test  and  Evaluation,  was 
generally  contractor  conducted  and  was  totally 
developmental  in  nature.  Air  Force  Regulation 
80-14  defined  CAT  I as  “development  testing 
and  evaluation  of  the  individual  components, 
subsystems,  and  in  certain  cases  the  complete 
system.'  There  were  Air  Force  Preliminary 
Evaluations  but  these  too  were  development 
test  and  evaluation  oriented. 

In  Category  II  (CAT  II),  System  Devel- 
opment Test  and  Evaluation,  the  participants 
changed  somewhat  but  the  emphasis  did  not. 
The  Air  Force,  through  Air  Force  Systems 
Command  (AFSC),  had  the  lead  role  with  the 
contractor  still  heavily  involved  but  in  a sec- 
ondary role.  Air  Force  Regulation  80-14  de- 
fined CAT  II  as  “test  and  evaluation  spanning 
the  integration  of  subsystems  into  a complete 
system  in  as  near  an  operational  configuration 
as  practicable.”  The  using  command  became 
involved  during  CAT  II  by  assigning  the  Dep- 
uty Test  Director  and  by  participating  in 
“hands-on”  maintenance  with  the  contractor  at 


the  test  site.  However,  CAT  II  was  seldom 
“operational  in  nature.”'  The  production  deci- 
sion for  the  system  was  generally  made  during 
CAT  II. 

Category  III,  System  Operational  Test 
and  Evaluation,  was  the  responsibility  of  the 
using  command.  It  included  “...all  components, 
support  items,  personnel  skills,  technical  data 
and  procedures.. .under  as  near  operational  con- 
ditions as  practicable.”'  Since  the  tests  were 
conducted  on  the  production  item  after  delivery 
to  the  using  command,  “such  testing  was  to 
determine  how  best  to  operate  a system,  rather 
than  to  provide  data  for  decisionmakers  to  use 
in  determining  whether  or  not  to  acquire  a 
system.”’ 

Such  was  the  situation  when  the  Blue 
Ribbon  Defense  Panel  made  its  “Report  to  the 
President  and  the  Secretary  of  Defense”  on  1 
July  1970.  The  Executive  Summary  statement 
relative  to  operational  test  and  evaluation  was 
direct  and  to  the  point.  “Operational  test  and 
evaluation  has  been  too  infrequent,  poorly  de- 
signed and  executed,  and  generally  inade- 
quate.’” The  development  test  and  evaluation 
was  “well  understood  and  faithfully  execut- 
ed...(and).. .not  considered  to  be  a major  prob- 
lem area.”  The  report  went  on  to  point  out  that 
operational  test  and  evaluation  was  not  just 
physical  testing  but  that  “To  be  effective, 
OT&E  must  be  a total  process,  using  all  appro- 
priate methods  of  evaluation,  which  spans  the 
entire  cycle  of  a system  from  initial  requirement 
until  it  is  phased  out  of  the  operational  forces.”* 


Referring  specifically  to  the  Air  Force,  the 
panel  stated: 

“There  are  three  principal 
problems  with  Air  Force  OT&E, 
as  currently  done.  First,  opera- 
tional considerations  receive 
much  too  little  attention  in  Cate- 
gories I and  II.  Second,  the  opera- 
tional commands  responsible  for 
Category  III  and  Operational  Em- 
ployment Testing  lack  both  the 
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personnel  and  facilities  to  be  effec- 
tive. Finally,  all  of  the  categories 
are  too  duplicative  and  time- 
consuming.”* 

The  panel  made  sweeping  recommenda- 
tions on  organization  within  the  Office  of  the 
Secretary  of  Defense  (OSD).  The  most  signifi- 
cant recommendation,  from  the  testing  stand- 
point, concerned  the  independent  Operational 
Test  and  Evaluation  agencies.  The  wording  is 
somewhat  indirect  but  the  meaning  is  clear. 
“The  Secretary  of  Defense  should  communi- 
cate to  the  Military  Departments... his  convic- 
tion (that)  the  cause  of  effective  OT&E  is  best 
served  when  independent  OT&E  organizations 
report  directly  to  the  Chiefs  of  Service,  Service 
Secretaries  or  both.”' 

The  DOD  response  was  immediate  and 
positive.  The  then  Secretary  of  Defense,  The 
Honorable  Melvin  Laird  asked  the  Services  to 
forward  individual  proposals  for  meeting  oper- 
ational test  and  evaluation  requirements  to  him 
not  later  than  1 September  1970.’  The  results 
were  inconclusive  and  resulted  in  a memoran- 
dum from  the  then  Deputy  Secretary  of  De- 
fense David  Packard  on  11  February  1971.  He 
wanted  an  operational  test  and  evaluation 
“...agency  which  is  separate  and  distinct  from 
the  developing  command  and  which  reports  the 
results  of  its  T&E  efforts  directly  to  the  Chief  of 
Service.”* 

The  Air  Force  response  did  not  establish 
the  independent  organization  but  it  did  make 
significant  changes.  The  concept  of  categories 
was  eliminated  and  replaced  with  the 
DT&E/OT&E  philosophy.  Also  introduced 
was  initial  operational  test  and  evaluation  to  be 
conducted  prior  to  the  production  decision. 
Active  participation  by  the  using  and  support- 
ing commands  was  incorporated  and  the  user 
was  required  to  provide  an  initial  operational 
test  and  evaluation  report  to  the  production 
decision  process.' 

Parallel  with  the  memorandum-response 
cycle  discussed,  the  first  DOD  Directive 
5(XX).l,  “Acquisition  of  Major  Defense  Sys- 
tems,” dated  July  31,  1971,  was  prepared  and 
released.  The  purpose  of  DODD  5(X)0. 1 was  to 


provide  formal  acquisition  policy  within  DOD, 
to  establish  the  Defense  Systems  Acquisition 
Review  Council  (DSARC)  and,  to  require  the 
Development  Concept  Paper  (DCP)  for  major 
programs.  Mention  of  direct  user  involvement 
prior  to  DSARC  II  (Full-Scale  Development 
decision)  was  not  made  nor  was  there  any 
requirement  for  an  independent  operational  test 
agency.  Note  that  this  directive  followed  by  5 
months  Secretary  Packard’s  memorandum  call- 
ing for  an  independent  agency.  However,  the 
Directive  was  explicit  on  the  role  and  impor- 
tance of  the  operational  assessment; 

“Test  and  evaluation  shall  commence 
as  early  as  possible.  A determination  of 
operational  suitability,  including  logis- 
tic support  requirements,  will  be  made 
prior  to  large-scale  production  com- 
mitments, making  use  of  the  most  real- 
istic test  environment  possible  and  the 
best  representation  of  the  future  opera- 
tional system  available.  The  results  of 
this  operational  testing  will  be  evalu- 
ated and  presented  to  the  DSARC  at 
the  time  of  the  production  decision.”* 

The  Air  Force  had  already  implemented 
both  the  spirit  and  the  intent  of  DODD  5000. 1 
through  a Chief  of  Staff  letter  on  Operational 
Test  and  Evaluation  dated  1 Apr  71  which 
called  for  “operational  test  and  evalua- 
tion...conducted  by  operational  forces  to  deter- 
mine operational  effectiveness  and  suitabili- 
ty...”’ The  reporting  channel  for  the  test  results 
was  independent  of  the  developer  and  the  Air 
Force  felt  that  this  complied  with  the  intent  of 
the  DOD  guidance  and  the  Blue  Ribbon  De- 
fense Panel  recommendations. 

Although  the  essence  of  the  policy  had 
been  directed  by  the  Deputy  Secretary  of  De- 
fense, Mr.  Packard,  in  a memorandum  dated  1 1 
February  1971,  it  was  not  until  19  January 
1973  that  specific  DOD  policy  on  test  and 
evaluation  was  formalized  into  DODD  5(XX).3, 
“Test  and  Evaluation”.  The  Directive  con- 
firmed the  existing  policy  that  operational  test 
and  evaluation  would  be  conducted  by  “opera- 
tional and  support  personnel”.’  Initial  Opera- 
tional Test  and  Evaluation  was  to  be  accom- 
plished during  Full-Scale  Development  to  assist 
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in  evaluating  operational  effectiveness.  The 
most  significant  requirement  was  that  “one 
major  field  agency  (or  a limited  number  of  such 
major  field  agencies)  separate  and  distinct  from 
the  developing/procuring  command”  be  estab- 
lished. This  agency  was  to  report  directly  to  the 
Chief  of  Staff  and  “Insure  that  the  OT&E  is 
effectively  planned  and  conducted.”’  The  oper- 
ational test  and  evaluation  was  to  be  separate 
from  development  test  and  evaluation  but  ini- 
tial operational  test  and  evaluation  could  be 
combined  with  development  test  and  evaluation 
if  “separation  would  cause  delay  involving 
unacceptable  military  risk,  or  would  cause  an 
unacceptable  increase  in  the  acquisition  cost”.’ 
In  either  eventuality,  the  operational  test  and 
evaluation  was  responsible  for; 

• insuring  that  the  tests  were  planned 
and  executed  so  as  to  provide  the  nec- 
essary operational  test  data; 

• participating  actively  in  the  tests  and, 

• separately  evaluating  the  test  results. 

There  was  no  mention  of  the  seeming  conflict 
with  DODD  5000. 1 that  stated  “The  develop- 
ment and  production  of  a major  defense  system 
shall  be  managed  by  a single  program  man- 
ager...” and  the  split  test  and  evaluation  respon- 
sibility specified  in  DODD  5000.3. 


AFTEC  individual  was  assigned  to  the  test 
force  and  given  operational  control  over  the 
using  command  personnel  who  were  already  in 
place.  This  action  created  considerable  conster- 
nation and  misunderstanding  of  the  AFTEC 
role.  Joint  Test  Force  (JTF)  Directors  per- 
ceived the  JTF  “test  teams”  being  fragmented 
and  the  Director’s  authority  diluted.  It  was 
difficult  if  not  impossible  to  draw  a Joint  Test 
Force  organizational  chart  that  really  depicted 
the  organization.  A strong  adversary  relation- 
ship existed  as  a result  of  the  new  organization 
being  imposed  upon  already  existing  and 
smoothly  running  test  forces. 

For  nearly  4 1/2  years,  DODD  5(XX3. 1 
went  unchanged,  while  DODD  5(XX).3  experi- 
enced only  minor  changes.  The  minor  changes 
to  the  latter  directive  deleted  the  option  of 
having  more  than  one  independent  agency  and 
further  stipulated  that  the  one  remaining  field 
agency  would  be  independent  of  the  using  com- 
mand and  the  developing  command.  The  22 
Dec  1975  change  to  DODD  5000.1  left  intact 
the  concept  of  the  single  Program  Manager 
being  responsible  for  development  of  the  system 
and  the  paragraph  on  test  and  evaluation  was 
repeated  verbatim.  Changes  were  not  made  to 
the  document  that  would  indicate  a shift  in  test 
and  evaluation  policy  from  the  emphasis  on 
early  development  test  and  evaluation  and  ini- 
tial operational  test  and  evaluation.” 


The  Air  Force  established  the  Air  Force 
Test  and  Evaluation  Center  (AFTEC)  as  a 
separate  operating  agency  on  1 January  1974. 
The  Air  Force  was  the  last  of  the  three  services 
to  initiate  an  independent  operational  test 
agency.  With  approximately  200  personnel, 
AFTEC  was  to  assess  the  operational  effective- 
ness, suitability  and  logistic  supportability  of 
new  Air  Force  systems  and  to  report  the  results 
directly  to  the  Air  Force  Chief  of  Staff.  The 
using  command  would  continue  to  provide  per- 
sonnel to  actually  conduct  the  operational  tests 
under  AFTEC  management.  Thus  the  indepen- 
dent evaluating  and  reporting  functions  were 
established  but  did  not  result  in  a significant 
change  in  the  number  of  persons  assigned  nor  in 
the  tasks  performed  by  these  persons.  The  only 
visible  change  at  the  test  sites  was  that  an 


CURRENT  OT&E  POLICY 

DOD  DIRECTIVE  5000.1 

Major  changes  occurred  approximately  1 
year  after  the  December  1975  . change  to 
DODD  5000.1  when  the  new  DODD  5000.1 
titled  “Major  Systems  Acquisition”  was  issued 
(18  January  1977).  Its  purpose  was  to  imple- 
ment OMB  Circular  A-109  dated  April  5,  1976 
also  titled  “Major  Systems  Acquisition,”  and  to 
update  DOD  policy  on  acquisition  manage- 
ment. Considerable  discussion  on  the  role  of  the 
Program  Manager  is  presented  in  DODD 
5000.1.  The  Directive  serves  to  further  empha- 
size the  Program  Manager’s  overall  responsibil- 
ity for  the  development  of  the  system.  The  test 
and  evaluation  paragraph  had  two  changes. 
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Without  defining  the  terms,  the  Directive 
added  the  requirement  for  “An  estimate  of  mili- 
tary utility  and  of  operational  effectiveness...” 
to  those  of  operational  suitability  and  logistic 
support  requirements."  Significantly,  the  new 
test  and  evaluation  paragraph  deleted  the  re- 
quirement that  the  operational  test  results  be 
presented  to  the  DSARC  at  the  time  of 
production  decision.  The  former  change  ap- 
pears to  be  a direct  quote  from  the  January 
1973  DODD  5000.3  and  is  not  significant.  The 
latter  was  clarified  in  a subsequent  interview 
with  Director  of  Defense  Research  and  Engi- 
neering (Test  and  Evaluation)  officials.  The 
change  reflects  the  decentralization  trend  called 
for  in  OMB  Circular  A- 109.  The  initial  opera- 
tional test  and  evaluation  results  are  now  to  be 
reported  to  the  Service  SARC  which  is  to  be 
chaired  by  the  Service  Secretary  or  Undersecre- 
tary. The  Secretary  then  will  report  the  com- 
bined DT&E/OT&E  results  to  the  DSARC  as 
appropriate. 

DOD  DIRECTIVE  5000.3 

While  the  new  DODD  5(XX).  1 has  little 
impact  on  test  and  evaluation,  the  subsequent 
revision  of  DODD  5000.3  contains  major  pol- 
icy changes.  Some  of  these  changes  appear  to 
stem  from  the  April  2,  1974  Defense  Science 
Board  “Report  of  Task  Force  on  Test  & 
Evaluation.”"  In  testimony  before  the  Re- 
search and  Development  Subcommittee  of  the 
House  Armed  Services  Committee  on  16 
March  1977,  General  Lotz*  stated  “...we  are 
making  changes  in  our  test  and  evaluation  poli- 
cies and  directives  to  emphasize  the  need  for 
conducting  operational  testing  earlier  in  the 
development  cycle  of  a weapons  system.”  Gen- 
eral Lotz  went  on  to  state  four  reasons,  para- 
phased  below,  for  the  need  for  earlier  opera- 
tional test  and  evaluation. 

a.  “In  considering  system  concepts 
during  the  early  stages  of  a program, 
we  need  to  know  not  only  whether  they 
are  technically  feasible  but  also 


•Lt  General  Walter  F..  Lotz,  Jr.,  USA  (Ret),  then  Deputy 
Director  (T&E),  Office  of  the  Director  of  Defense  Re-  ' 
search  and  Engineering  (ODDR&E). 


whether  they  are  operationally  viable 
or  tactically  sound. 


b.  To  have  any  impact  on  the  research 
and  development  phase  of  a program, 
OT&E  should  be  conducted  much 
earlier  than  at  DSARC  III  since  by 
then  most  R&D  resources  will  have 
been  expended. 


c.  The  earlier  in  the  development  cycle  a 
deficiency  is  identified  and  corrected, 
the  less  the  cost  of  corrective  action. 


d.  A trend  toward  initiating  limited  pro- 
duction before  the  major  production 
decision  results  in  a commitment  of 
procurement  funds  prior  to  the  major 
production  decision.”" 


The  author  was  privileged  to  review  the  1 2 
July  1977  draft  revision  to  DOD  Directive 
5(XX).3,  that  shows  the  current  thrust  of  OSD 
thought,  although  the  revision  may  be  further 
modified  during  the  review  and  coordination 
process.  An  analysis  of  the  draft  document 
revealed  the  following  significant  changes  envi- 
sioned relative  to  planning,  timing  and  report- 
ing of  operational  test  and  evaluation.  In  gen- 
eral, the  directive  would  require  that  the  ”... 
critical  issues,  test  criteria  and  measures  of 
effectiveness  related  to  the  satisfaction  of  mis- 
sion need  shall  be  established  prior  to  the  com- 
mencement of  tests.”"  Further,  the  directive  re- 
quires the  establishment  of  a management  re- 
serve both  in  schedule  and  funds  to  cover  con- 
tingency testing  if  significant  test  objectives 
have  not  been  met."  The  term  “management 
reserve”  is  not  used  but  the  intent  is  clear. 


Specifically,  operational  test  and  evalua- 
tion will  be  directly  related  to  the  modified 
milestone  concept  and  the  requirements  to  as- 
sess the  system’s  vulnerability  and  its  capability 
against  enemy  countermeasures  will  be  added. 
The  1973  policy  did  not  call  for  operational  test 
and  evaluation  prior  to  the  full-scale  develop- 
ment phase.  The  1977  policy  requires  that  ini- 
tial operational  assessment  begin  in  the  concep- 
tual phase  following  Milestone  0,  to  assess  the 
operational  impact  of  the  proposed  technical 
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approaches.  During  the  demonstration  and  val- 
idation phase,  initial  operational  test  and  evalu- 
ation would  “...provide  information  relative  to 
projected  operational  effectiveness  and  suitabil- 
ity of  the  candidate  system.”'*  Significantly,  ini- 
tial operational  test  and  evaluation  results  may 
(as  opposed  to  wUI)bc  considered  at  Milestone 
II  as  may  any  commitment  of  funds  for  items 
having  a long  lead  time  or  limited  production 
items.  The  previously  specified  full-scale  engi- 
neering development  phase  involvement  re- 
mains the  same  except  that  the  initial  opera- 
tional test  and  evaluation  report  to  support  the 
production  decision  will  be  made  to  the  Service 
SARC  instead  of  the  DSARC. 

To  determine  what  portion  of  develop- 
ment test  and  evaluation  will  contribute  to  the 
accomplishment  of  operational  test  and  eval- 
uation objectives,  the  Air  Force  Test  and  Eval- 
uation Center  is  now  directed  to  participate  in 
development  test  and  evaluation  planning. 
Early  coordination  should  serve  to  reduce  du- 
plication of  effort,  minimize  required  test  re- 
sources and  provide  maximum  data  to  satisfy 
common  needs  of  the  developer  (Air  Force 
Systems  Command)  and  AFTEC. 


is  definitely  shifting  toward  the  combined  test- 
ing concept  being  regarded  as  the  most  efficient 
and  cost  effective  method  of  test  and  evalua- 
tion. The  key  is  to  ensure; 

• that  the  combined  tests  are  planned 
and  executed  to  realistically  provide 
the  necessary  test  information; 

• that  AFTEC  participates  actively;  and, 

• that  AFTEC  conducts  a separate  eval- 
uation of  the  test  data. 

The  1974  Defense  Science  Board  report 
stated  that  lOT&E  “tests  should  not  be  con- 
ducted until  the  primary  objectives  of  the 
DT&E  have  been  met.”'*  In  an  apparent  partial 
response,  the  new  DODD  5000.3  calls  for 
AFTEC  to  “Monitor  and  review  the  results  of 
DT&E.. .to  assess  the  readiness  of  the  system 
for  operational  testing.”'*  Close,  early  involve- 
ment of  AFTEC  in  the  acquisition  process 
should  reduce  the  inevitable  debate  between  the 
engineer  who  is  never  finished  and  the  opera- 
tional tester  who  is  anxious  to  start. 


A subtle  change  in  words  relative  to  com-  There  is  an  increased  emphasis  placed  on 

bined  development  test  and  evaluation  and  ini-  the  timely  preparation  of  the  Test  and  Evalua- 

tial  operational  test  and  evaluation  has  put  a tion  Master  Plan  (TEMP).  Up  to  now  it  had  to 

more  realistic  emphasis  on  the  issue  in  these  be  prepared  "prior  to  initiation  of  Full-Scale 

times  of  tight  money.  The  1973  directive  stated  Development”  and  was  to  “integrate  the  effort 

that  “...operational  testing  should  be  separate  and  schedules.”  Now  the  TEMP  must  be  pre- 

from  development  testing.  However,. ..(they)...  pared  as  early  as  possible  in  the  acquisition 

may  be  combined  where  separation  would  process  and  “...should  identify  and  integrate 

cause  delay,. ..or  would... increase  the  acquisi-  objectives,  responsibilities,  resources,  and 

tion  cost  of  the  system.”’  The  new  directive  schedules  for  all  test  and  evaluation  to  be  ac- 

modifies  the  statement  to  say  “Development  complished...”'*  This  requirement  alone  should 

testing  and  operational  testing  may  be  com-  result  in  earlier  involvement  of  AFTEC  because 

bined....”'*  The  remainder  of  the  words  are  es-  without  AFTEC  participation  the  TEMP 

sentially  the  same  but  the  emphasis  is  shifted  to  would  be  incomplete.  The  same  emphasis  is  put 

combined  testing  as  combined  testing  is  stated  on  the  test  portion  of  the  DCP  (now  called 

first.  Interviews  with  DDR&E  officials  con-  Decision  Coordinating  Paper).  The  Air  Force 

firmed  the  intent  of  the  change.  However,  a sen-  Test  and  Evaluation  Center  inputs  will  be  re- 

tence  added  at  Navy  insistence  clouds  the  issue  quired  to  prepare  a complete  Decision  Coordi- 

slightly:  “As  a normal  practice  the  operational  nating  Paper, 

tests  supporting  a production  decision  will  be 

conducted  independently  by  the  OT&E  agen-  The  reader  is  reminded  that  the  foregoing 

cy.”'*  Despite  this  add-on,  the  availability  of  re-  assessment  of  the  draft  DOD  Directive  5000.3 

sources,  time  and  cost  considerations  will  re-  may  have  to  be  modified  if  mgjor  alterations  in 

main  the  final  determinents.  But  the  emphasis  the  draft  occur  before  Hnal promulgation. 
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AIR  FORCE  REGULATIONS 

Air  Force  policy  on  test  and  evaluation 
has  not  had  time  to  react  to  the  new  DOD 
policies.  The  Air  Force  test  and  evaluation 
policy  regulation,  AFR  80-14,  and  the  AFTEC 
regulation,  AFR  23-36,  are  both  mid- 1976  doc- 
uments. The  most  current  element  is  the  Air 
Force  Systems  Command  Supplement  to  AFR 
80-14,  dated  3 January  1977. 

Two  key  elements  of  the  current  AFR 
80-14  are  the  relationship  between  the  Pro- 
gram Manager  and  AFTEC  and  the  conduct  of 
combined  testing.  “The  Program  Manager  has 
overall  responsibility  for  a system  acquisition 
program  (except  the  management  of  OT&E.)”“ 
The  Program  Manager  will  include  operational 
test  and  evaluation  requirements  in  the  test  pro- 
gram and  support  operational  test  and  evalua- 
tion as  appropriate.  “AFTEC  has  responsibility 
for  managing  the  OT&E  in  a major  acquisition 
program. ..AFTEC  will  plan,  direct,  conduct, 
control,  and  independently  evaluate  and  report 
on  OT&E.”'‘ 

Paragraph  17  of  AFR  80-14,  “Conduct- 
ing a Combined  Test  Program”  seems  to  negate 
any  sort  of  team  concept  in  combined  testing. 
The  planning  aspects  of  the  paragraph  conflict 
with  the  new  DODD  5000.3  in  that  these  plan- 
ning aspects  call  for  separate  development  test 
and  evaluation  and  operational  test  and  evalua- 
tion test  plans  with  the  Program  Manager  re- 
sponsible for  integrating  the  operational  test 
and  evaluation  plan.  Joint  planning  is  not  men- 
tioned. The  separation  of  roles  and  functions 
continues  down  to  the  Joint  Test  Force.  The 
test  force  director,  appointed  by  Air  Force 
Systems  Command,  is  responsible  for  develop- 
ment test  and  evaluation,  integrating  combined 
test  events,  insuring  availability  of  resources 
and  insuring  the  safety  of  the  test  program.  The 
operational  test  and  evaluation  test  director, 
provided  by  AFTEC,  manages  the  operational 
test  and  evaluation  portion  of  the  combined 
program.  Previously  this  function  was  per- 
formed by  the  user  detachment  commander 
serving  as  Deputy  Joint  Test  Force  Director  as 
opposed  to  “OT&E  test  director.”  Thus  there  is 
one  Joint  Test  Force  with  two  directors — a 


situation  that  tends  to  splinter  the  “test  team” 
and  is  difficult  to  depict  on  an  organizational 
chart.  Air  Force  Systems  Command  Supple- 
ment 1 to  AFR  80-14  attempts  to  improve  the 
situation  by  stating  that  "This  test  force  must 
be  able  to  function  efficiently  as  an  entity  to 
accomplish  overall  program  test  objectives; 
management  relations  must  be  clearly  defined" 
(underlining  added  for  emphasis).”  Unfortu- 
nately  neither  AFR  80-14  nor  AFSC  Supple- 
ment 1 specifies  one  individual  who  has  com- 
plete test  program  responsibility.  Air  Force 
Regulation,  AFR  23-36“  generally  follows  the 
guidance  set  forth  in  AFR  80-14.  Appropriate 
Air  Staff  agencies  are  aware  of  the  difficulties 
with  the  combined  test  program  guidance  and 
hopefully  will  consider  again  this  combined 
guidance  in  the  next  revision  of  AFR  80-14. 

The  Air  Force  Test  and  Evaluation  Center 
has  been  in  existence  for  more  than  3 years.  Ini- 
tially there  were  serious  growing  pains  as  this 
new  organization  was  imposed  upon  ongoing 
programs.  An  adversary  had  been  thrown  into 
a predominantly  advocate  environment.  Very 
few  individuals  really  understood  the  role  of  the 
independent  test  agency.  The  situation  has  now 
had  time  to  stabilize  and  AFTEC  is  fully  opera- 
tional. In  fact,  AFTEC  recently  was  authorized 
sixty  additional  personnel.  The  AFTEC  FY 
1978  budget  calls  for  $17.9  million  in  3080 
(Other  Procurement),  3400  Operation  and 
Maintenance  (O&M)  and  3600  Research  De- 
velopment Test  and  Evaluation  (RDT&E) 
money.  This  is  more  than  double  the  $8.46  mil- 
lion FY  1976  budget  and  the  size  of  the  budget 
is  expected  to  continue  to  rise.  Figures  were  not 
available  on  the  cost  of  AFTEC  managed  oper- 
ational test  and  evaluation  but  it  can  be  as- 
sumed to  be  significant  since  approximately  1 5 
percent  of  the  annual  DOD  Research  Develop- 
ment Test  and  Evaluation  cost  for  weapons  sys- 
tems is  used  for  test  and  evaluation.” 

COMPARATIVE  VIEWS  ON  OT&E 

With  the  relative  significance  of  AFTEC 
and  operational  test  and  evaluation  established, 
a series  of  structured  interviews  was  conducted 
to  determine  the  current  view  of  operational 
test  and  evaluation  in  the  Air  Force.  Through 
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these  interviews,  in  which  eight  questions  were 
asked,  the  views  of  various  Department  of  De- 
fense and  Air  Force  officials  were  compared 
with  those  of  the  AFTEC  Commander.  Offi- 
cials from  both  development  test  and  evalua- 
tion and  operational  test  and  evaluation  orga- 
nizations were  interviewed  as  well  as  senior 
DD(T&E)  officials. 

The  approach  used  to  analyze  the  inter- 
views was  to  determine  if  a consensus  of  opin- 
ion existed  on  the  issues.  Next  the  results  were 
compared  with  the  AFTEC  view.  Note  that  the 
interviewees  were  expressing  personal  opinions 
and  not  necessarily  Air  Force  nor  Department 
of  Defense  policy. 

Question  1. 

DODD  5000. 1 calls  for  a “strong  SPO 
to  achieve  program  objectives.”  AFR 
80-14  says  that  the  Program  Manager 
has  “overall  responsibility”  except  for 
Operational  Test  and  Evaluation.  Does 
this  in  effect  dilute  the  Program  Man- 
ager’s authority? 

With  minor  reservations,  the  consensus  is  that 
significant  problems  with  this  concept  do  not 
exist  at  this  time.  The  view  is  that  since  the 
Program  Manager  has  control  of  the  resources 
of  the  program,  the  dollars  and  the  test  articles, 
he  is  in  a strong  negotiating  position  when  the 
original  test  plans  are  being  developed.  The 
negotiating  task  is  a delicate  one  that  requires 
diplomacy  and  tact  in  establishing  effective 
working  relationships  with  AFTEC.  The  same 
reasoning  also  applies  at  the  test  force  level 
where  the  split  roles  are  evident.  The  Joint  Test 
Force  Director  has  the  necessary  authority 
through  his  “ownership”  of  the  test  articles  and 
responsibility  for  safety  of  operations.  The  con- 
cept of  independence  is  important  but  not  at 
overall  program  expense.  In  the  final  analysis 
the  golden  rule  applies:  “He  who  has  the  gold, 
rules.”  Therein  lies  the  authority  that  the  Pro- 
gram Manager  needs  and  that  can  be  used  as 
appropriate. 

The  AFTEC  Commander,  Major  General 
Leaf,  views  this  issue  in  a similar  fashion. 


Rather  than  viewing  the  Program  Manager’s 
authority  as  being  diluted.  General  Leaf  sees 
the  situation  as  merely  an  affirmation  of  the 
differing  roles  of  development  test  and  evalua- 
tion and  initial  operational  test  and  evaluation, 
the  former  being  specification  oriented  and  the 
latter  being  oriented  toward  the  operational 
environment.  Furthermore,  because  of  the 
complementary  nature  of  development  test  and 
evaluation  and  initial  operational  test  and  eval- 
uation, the  Program  Manager’s  ability  to 
produce  an  operationally  effective  and  suitable 
system  is  enhanced. 

As  AFTEC  has  matured  and  the  rest  of 
the  Air  Force  has  gained  a better  understanding 
of  its  role,  significant  conflicts  in  this  area  have 
diminished.  However,  the  issue  should  be  put 
into  clearer  context  in  a subsequent  change  to 
DODD  5000.1,  the  basic  acquisition  manage- 
ment directive,  and  AFR  80-14.  Further  defini- 
tion would  preclude  problems  from  reoccurring 
when  the  personalities  involved  are  rotated  to 
other  assignments. 

Question  2. 

How  much  should  initial  operational 
test  and  evaluation  requirements  influ- 
ence total  Full-Scale  Engineering  De- 
velopment test  schedules  and  the  Mile- 
stone III  date? 

There  is  no  clearcut  consensus  on  this 
question.  Since  all  programs  are  different,  the 
effect  will  be  different.  The  common  thread 
throughout  the  responses  was  that  well 
planned,  achievement  oriented  programs  (vs 
time-schedule  oriented  programs)  would  not 
have  problems  with  the  DSARC  process.  There 
is  a danger  of  unnecessary  program  stretchout 
if  the  test  program  is  not  well  structured  since 
the  requirement  for  an  initial  operational  test 
and  evaluation  input  to  the  production  decision 
process  is  a firm  one.  Both  the  Program  Man- 
ager and  AFTEC  must  realize  that  initial  oper- 
ational test  and  evaluation  is  not  an  end  in  itself 
but  rather  it  is  one  of  the  means  to  the  end  ob- 
jective of  selecting  the  correct  production  deci- 
sion. When  the  areas  of  risk  have  been  ade- 
quately addressed,  go  for  production,  don ’t  con- 
tinue to  test  to  obsolescence. 
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Again  AFTEC  concurred  by  emphasizing 
that  adverse  influence  should  not  be  experi- 
enced if  the  initial  operational  test  and  evalua- 
tion requirements  are  integrated  into  the  pro- 
gram early  and  are  properly  reflected  in  the 
Statement  of  Work  and  the  contract. 

Question  3. 

Have  we  swung  from  too  little  to  too 
much  testing  (DT&E  or  lOT&E)  prior 
to  Milestone  III? 

There  is  wide  disagreement  on  this  ques- 
tion. Interestingly,  the  developer  oriented  re- 
sponses indicated  that  we  are  doing  too  much 
testing  or  at  least  leaning  that  way.  There  is  also 
a feeling  that,  regardless  of  the  amount  of 
testing,  the  areas  of  emphasis  need  to  be  scruti- 
nized. There  is  a concern  that  overemphasis  on 
initial  operational  test  and  evaluation  at  the 
expense  of  development  test  and  evaluation  is 
dangerous  and  that  developmental  “show  stop- 
pers” could  be  missed.  The  best  summation  in 
this  area  is  that  the  system  should  be  developed 
first  and  the  concept  of  combined  developmen- 
tal and  operational  testing  (with  independent 
evaluation)  be  used  to  the  maximum  extent 
practicable. 

The  AFTEC  response  quoted  the  1977 
Defense  Science  Board  report: 

“There  appears  to  be  little  or  no 
overtesting  done  under  the 
(DOD)  directives;  what  testing  is 
done  contributes  its  full  value  to 
the  improvement  and  verification 
of  system  performance.”'* 

Question  4. 

What  are  we  getting  now  in  terms  of 
operational  assessment  that  was  not 
available  prior  to  1974  when  Air  Force 
Systems  Command  and  the  users  were 
working  together?  In  other  words, 
what  new  contribution  is  AFTEC 
making? 


Questions. 

The  AFTEC  budget  projection  for  FY 
1978  is  $17.9  million  and  will  be  higher 
in  the  out  years.  Do  you  feel  that  we 
are  getting  our  money’s  worth? 


The  only  apparent  consensus  on  these  two 
questions  is  that  the  AFTEC  main  contribution 
is  its  independent  test  report  to  the  Chief  of 
Staff  and  to  the  AFSARC.*  Otherwise,  the 
responses  are  mixed.  Some  feel  that  AFTEC 
contributions  are  very  worthwhile  but  that  the 
cost  effectiveness  has  yet  to  be  proved.  Others 
feel  that  the  Air  Force  was  doing  Just  as  well 
prior  to  AFTEC  involvement.  Whether  or  not 
the  AFTEC  contribution  is  cost  effective  will 
not  be  known  until  someone  can  determine  a 
way  of  measuring  costs  avoided  because  of  the 
AITEC  input.  Any  deficiency  found  early 
enough  to  correct  during  production  instead  of 
through  retrofit  equates  to  costs  avoided.  In  the 
future,  it  may  be  possible  to  develop  a means  to 
compare  the  number  of  postproduction  changes 
before  AFTEC  inception  (1974)  and  after 
AFTEC.  Only  then  can  some  valid,  statistical 
measure  of  cost  effectiveness  be  made.  Until 
then,  the  AFTEC  contribution  is  recognized  as 
significant  in  the  Air  Force  and  at  Department 
of  Defense  and  Congressional  levels. 

The  AFTEC  response  reinforced  the  posi- 
tive aspects  of  operational  test  and  evaluation  in 
the  acquisition  process  by  citing  several  specific 
contributions.  In  addition  to  the  reporting  as- 
pect, the  contributions  include: 

a)  revision  of  the  deficiency  reporting 
process  to  provide  better  visibility,  “prioritiza- 
tion” of  deficiencies,  and  identification  of  po- 
tential fixes  through  coordination  with  the  de- 
veloper and  the  user: 

b)  standardization  of  operational  test  and 
evaluation  planning,  criteria,  execution  and 
support  as  well  as  evaluation  methodology; 
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c)  improved  software  evaluation  through 
the  use  of  a software  evaluation  group; 

d)  earlier  involvement  of  operational  test 
and  evaluation  personnel  at  the  program  office 
and  at  contractor  facilities;  (Note  that  user 
personnel  were  collocated  at  both  program  of- 
fice and  contractor  facilities  on  the  B-1  pro- 
gram prior  to  the  inception  of  AFTEC  in  1974.) 

e)  improved  logistics  and  manpower  as- 
sessments and  operations  and  support  cost 
estimates. 

From  a cost  effectiveness  (cost  avoidance) 
standpoint,  the  early  identification  of  problems, 
when  problems  are  relatively  cheap  to  fix,  is 
most  significant.  The  example  of  the  F-15 
production  radar  is  a case  in  point.  Major 
General  Leaf  pointed  out  that  the  APG-63 
production  radar  would  not  have  been  fixed 
without  AFTEC  having  surfaced  the  problem 
at  high  levels  in  the  Air  Force. 

Question  6. 

AFTEC  has  been  criticized  as  a ‘nay 

sayer’.  Is  this  a valid  criticism? 

Initially,  the  AFTEC  reports  seemed  ex- 
cessively negative  and  without  balance.  The 
interviewees  generally  agreed  that  this  had  been 
true  but  that  it  is  no  longer  a significant  factor. 
The  feeling  is  that  AFTEC  personnel  have 
matured  in  the  job  and  have  a better  under- 
standing of  the  AFTEC  role  as  does  the  rest  of 
the  Air  Force.  The  consensus  is  that  AFTEC 
should  be  objective — neither  success  nor  failure 
oriented — and  should  present  a balanced  re- 
port. There  remains  the  danger  of  the  initial 
operational  test  and  evaluation  comments  being 
taken  out  of  context  outside  of  the  Air  Force. 
This  latter  danger  makes  the  need  for  balanced 
objectivity  even  more  important. 

The  AFTEC  response  agrees  with  the 
above.  The  AFTEC  charter  calls  for  evaluation 
without  bias  or  prejudice.  The  objective  is  to 
provide  an  operational  assessment  that  high- 
lights both  the  positive  and  negative  aspects  of 
the  system  being  evaluated. 


Question  7. 

Does  it  really  matter  who  gathers  the 
flight  test  data  (developer  or  operator) 
if  the  test  conditions  are  operationally 
representative? 

This  has  been  an  emotional  issue.  In  some 
programs,  such  as  the  B-1  and  F-16,  the  “nego- 
tiations” reached  the  general  officer  level  to 
determine  the  required  mix.  In  an  unlimited 
resource  environment  there  would  not  be  an 
issue  since  initial  operational  test  and  evalua- 
tion and  development  test  and  evaluation  could 
be  clearly  separated.  In  combined  programs 
with  limited  resources,  the  mix  is  significant. 
The  responses  to  this  question  varied  from, 
“you  can’t  have  an  operational  test  if  it  is  all 
done  by  developers”  to  the  feeling  that  “a  data 
point  was  a data  point.”  Overall,  there  was 
agreement  that  the  operational  pilot  has  a dif- 
ferent perspective  than  that  of  the  trained  test 
pilot  who  has  been  out  of  the  operational  envi- 
ronment for  a number  of  years.  Qualitative 
assessment  by  both  developer  and  user  pilots 
are  required  for  a balanced  evaluation.  Another 
aspect  was  colorfully  tagged  the  “hamburger” 
factor.  Pilot  skills  vary.  Test  pilots  and  the  best 
operational  pilots  evaluating  a new  aircraft, 
might  subconsciously  compensate  for  an  air- 
craft characteristic  that  a pilot  of  lesser  skill 
could  not  find  acceptable.  Thus  there  is  a need 
for  a mix  of  pilots.  The  Program  Manager  and 
AFTEC  must  jointly  agree  on  the  proper  mix. 

From  the  standpoint  of  pure  performance 
and  fiying  qualities  data  taking,  AFTEC  does 
not  see  a requirement  to  specify  who  flies  the 
test  missions  since  the  missions  are  quantitative 
in  nature.  This  is  not  true  for  the  operational 
effectiveness  evaluation.  For  initial  operational 
test  and  evaluation,  the  user  input  is  manda- 
tory. Thus  AFTEC  concurs  with  the  overall 
consensus  of  opinion. 

Question  8. 

AFTEC  currently  is  limited  to  “man- 
aging” the  lOT&E  program.  Should  it 
also  have  the  capability  to  “conduct” 
lOT&E  as  does  its  Navy  counterpart? 
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The  unanimous  response  to  this  final 
question  was  “NO”.  Such  a concept  would 
inevitably  lead  to  another  “super  organization” 
or  “operational  monster”  that  the  Air  Force 
cannot  afford.  Such  concept  would  be  redun- 
dant and  would  absorb  MAJCOM  efforts  that 
rightly  belong  to  the  user  for  follow-on  opera- 
tional test  and  evaluation.  Further  studies  may 
offer  a better  system  than  currently  exists  but 
the  current  concept  is  considered  much  better 
for  the  Air  Force  than  is  the  Navy  approach  of 
a separate  operational  test  and  evaluation  force. 

Not  surprisingly,  the  AFTEC  response 
was  in  complete  agreement  with  the  unanimous 
response  noted  above.  It  was  pointed  out  that 
the  current  concept  is  in  concert  with  the 
planned  new  DODD  5000.3.  Since  the  AFTEC 
test  team  personnel  are  “borrowed”  from  the 
using  commands  the  team  is  current  in  opera- 
tional matters — more  so  than  if  the  team  mem- 
bers were  assigned  to  an  independent  agency. 
There  is,  as  a result,  less  danger  of  AFTEC 
personnel  becoming  “professional  testers”  and 
losing  sight  of  the  AFTEC  role.  Furthermore, 
the  sharing  of  test  resources  fosters  combined 
DT&E/IOT&E  testing  which  is  inherently 
more  cost  effective  if  properly  done. 

INCORPORATING  OPERATIONAL 
TEST  AND  EVALUATION  INTO  THE 
PROGRAM 

Regardless  of  the  variety  of  opinion  on  the 
utility  and  effectiveness  of  AFTEC  in  the  ac- 
quisition process,  the  fact  remains  that  AFTEC 
is  here.  The  AFTEC  contribution  is  highly  visi- 
ble and  time  sensitive  in  the  decision  process. 
However,  in  the  final  analysis,  it  is  the  Program 
Manager  who  is  held  responsible  for  the  pro- 
gram outcome.  How  effectively  the  Ks/zid opera- 
tional requirements  are  incorporated  into  the 
program  will  be  a m.ajor  determinent  of  the  pro- 
gram outcome.  What  can  the  Program  Man- 
ager do  to  ensure  that  all  valid  operational  re- 
quirements are  adequately  covered  in  the  pro- 
gram? 

The  answer  to  this  extremely  important 
question  can  be  broken  into  three  major  areas: 


• Establishing  the  baseline 

• Organizing 

• Planning 

Establishing  the  Baseline 

Establishing  the  baseline  involves  a num- 
ber of  elements.  First  of  all,  know  the  signifi- 
cant DOD  Directives  and  AF  Regulations — 
DODD  5000.1  for  overall  guidance;  DODD 

5000.2  for  the  Milestone  checklists  and  Deci- 
sion - Coordinating  Paper  content;  DODD 

5000.3  for  the  specific  DOD  test  and  evaluation 
policy;  and,  AFR  80-14  and  AFSC  Supp  1 for 
Air  Force  test  and  evaluation  policy.  Also,  “A 
Guide  to  Program  Management”,  AFSCP 
800-3,  is  an  excellent  reference. 

Second,  know  the  operational  concept  for 
the  system.  The  operational  concept  is  the  key 
baseline.  It  has  to  be  included  in  the  system 
design  as  well  as  the  test  plan.  This  concept  will 
serve  as  the  criterion  against  which  the  system 
effectiveness  is  measured.  A new  Air  Force 
policy  for  initiating,  developing,  updating  and 
approving  operational  concepts  for  new  and 
improved  major  systems  is  being  developed  and 
the  Program  Manager  has  to  be  involved  if  the 
system  under  development  is  to  meet  the  need. 
Figure  1 depicts  the  flow  of  the  operational 
concept.  “ The  Required  Operational  Capabil- 
ity (ROC)*  has  always  been  part  of  the  system 
documentation.  The  ROC  leads  to  the  Mission 
Element  Need  Statement  (MENS)  which  is  a 
new  requirement  under  DODD  5000.1.  The 
Program  Manager  needs  to  be  intimately  famil- 
iar with  both  the  ROC  and  the  MENS.  The 
proposed  operational  concept  will  be  a dynamic 
document  tied  to  program  milestones  for  up- 
date and  refinement.  Air  Force  Systems  Com- 
mand supports  this  concept  and  the  require- 
ment for  approval  of  operational  concept 
changes  to  preclude  inadvertant  breaches  of 
program  thresholds.  The  concept  format  con- 
tains operational  thresholds,  forces,  standards 
for  development,  organization,  basing  and  sup- 
port, and  significantly,  the  updated  baseline  for 


'Recently  renamed  General  Operational  Requirement 
(GOR). 
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system  design  and  test  and  evaluation.  The 
benefits  for  the  Program  Manager  from  this 
concept  are: 

• that  it  will  provide  a forum  for  im- 
proved dialogue  between  the  program 
office,  the  user,  supporter  and  AFTEC; 

• that  it  will  improve  the  means  to  en- 
sure that  the  system  design  reflects  the 
operational  concept;  and, 

• that  it  will  ensure  corporate  reviews  of 
changes. 

“The  corporate  inputs  to  the  initial  and  subse- 
quent updates  of  the  operational  concept  pro- 
vide the  user,  developer,  supporter  and  tester  a 
voice  in  the  operational  concept.  At  initiation 
and  each  update,  the  operational  concept  docu- 
ment is  required  to  go  to  all  applicable  agencies 
for  review  and  comment  prior  to  being  submit- 
ted by  the  user  to  Headquarters  USAF  for 
approval.  The  proposed  operational  concept 
directive  will  provide  a means  to  ensure  that  the 
operational  community  complements  the  sys- 
tem acquisition  process  and  help  ensure  that  we 
get  the  best  system  for  the  money.”“  Whether 
or  not  the  above  system  becomes  official  Air 
Force  policy,  it  is  the  type  of  approach  that  the 
Program  Manager  should  use  to  ensure  that  all 
parties  concerned  with  his  system  are  working 
from  the  same  baseline. 

Third,  establish  the  critical  questions  and 
issues  to  be  addressed  in  the  test  program.  This 
will  require  coordination  with  such  organiza- 
tions as  Deputy  Director  for  Research  and  En- 
gineering (Test  and  Evaluation),  the  Air  Force 
Deputy  Chief  of  Staff  for  Research  and  Devel- 
opment, the  Air  Force  Deputy  Chief  of  Staff  for 
Operations,  Air  Force  Systems  Command,  and 
the  AFTEC/Using  Command.  The  critical 
questions  will  be  the  questions  that  the  Pro- 
gram Manager  will  have  to  address  at  the  pro- 
gram Milestones  to  gain  approval  to  proceed  to 
the  next  phase  of  the  development  cycle.  The 
questions  and  issues  will  be  reflected  in;  the 
Decision  Coordinating  Paper  (refer  to  DODD 
5000.2);  the  Test  and  Evaluation  Objectives 
Annex  (TEOA)  of  the  Program  Management 


Directive  (refer  to  AFR  80-14);  the  Test  and 
Evaluation  Master  Plan  (TEMP)  (refer  to  AFR 
80-14/AFSC  Supp  1);  and,  the  Program 
Management  Plan  (PMP)  (refer  to  AFSC  Pam- 
phlet 800-3). 

Organizing 

The  Program  Manager  cannot  accomplish 
all  of  the  above  tasks  without  the  assistance  of  a 
strong  organization.  The  key  man  will  be  the 
Program  Director  of  Test  and  Evaluation.  For 
an  aircraft  program,  this  individual  should  be  a 
qualified  test  pilot  who  is  current  in  aircraft 
similar  to  the  type  being  developed.  The  posi- 
tion should  be  rated  and  require  attendance  at 
the  Executive  Refresher  Course  at  the  Defense 
Systems  Management  College.  The  test  office 
should  be  manned  sufficiently  to  be  the  pro- 
gram’s single  voice  of  test  and  should  be  respon- 
sible for  all  ground  and  flight  tests  throughout 
the  program  to  preclude  conflicting  guidance  to 
the  contractor. 

Planning 

With  the  test  office  as  the  core,  the  Pro- 
gram Manager  should  form  the  Test  Planning 
Working  Group  in  accordance  with  AFR 
80-14  and  AFSC  Supp  1.  This  group  should 
include  all  agencies  that  will  be  involved  in  the 
test  program;  the  responsible  test  organization 
(RTO);  AFTEC;  the  using  and  supporting 
commands;  and  the  system  contractor  if  identi- 
fied. The  Test  Planning  Working  Group  con- 
cept worked  well  in  the  past  before  it  was 
required  by  AFR  80-14.  The  concept  was  used 
extensively  and  very  successfully  in  the  B-1  test 
program.  The  Group  provided  for  the  neces- 
sary forum  for  all  test  related  subjects  ranging 
from  establishing  objectives  and  baselines  to 
defining  organizational  responsibilities.  With 
the  combined  inputs  of  all  organizations  along 
with  the  operational  concept  and  critical  issues, 
the  Test  Planning  Working  Group  can  prepare 
the  draft  TEMP,  TEOA,  and  the  test  portion  of 
the  Program  Management  Plan  and  Decision 
Coordinating  Paper.  The  working  group  will  be 
involved  in  the  preparation  of  the  detailed  test 
plans  whether  or  not  the  contractor  is  assigned 
primary  planning  responsibility. 
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The  Program  Manager  has  to  be  directly 
involved  in  this  planning  process.  Based  on  the 
program  charter,  he  must  provide  specific  guid- 
ance to  the  Program  Test  Director.  While  the 
Program  Manager  is  responsible  for  integrating 
the  initial  operational  test  and  evaluation  re- 
quirements into  the  program,  he  should  not 
blindly  accept  them.  All  requirements  and 
changes  to  requirements  must  be  questioned  to 
verify  validity  and  need.  The  requirements 
should  be  listed  in  order  of  priority  as  specific 
objectives  rather  than  as  percentages  of  flying 
hours  or  missions.  The  Program  Manager  must 
ensure  that  the  aircraft  is  adequately  developed 
through  the  accomplishment  of  orderly  devel- 
opment test  and  evaluation  objectives  before 
jointly  agreeing  with  AFTEC  that  the  system  is 
ready  for  initial  operational  test  and  evaluation. 
Finally,  the  Program  Manager  must  ensure 
that  the  output  of  the  planning  process  is  a 
jointly  developed  strategy  that  is  agreed  to  by 
all  parties  concerned.  Only  a coordinated,  pro- 
gram oriented  plan  can  ensure  that  there  will 
not  be  surprises  later  in  the  program  because  of 
a previously  unidentified  requirement  that  is 
either  out  of  scope  or  is  one  that  severely 
impacts  the  schedule.  Finally,  the  Program 
Manager  will  have  complied  with  both  the 


spirit  and  the  intent  of  DODD  5000.3  through 
the  early  incorporation  of  the  initial  operational 
test  and  evaluation  requirements. 

SUMMARY 

The  criticisms  of,  and  changes  in.  Depart- 
ment of  Defense  and  Air  Force  policy  on  opera- 
tional test  and  evaluation  from  1970  through 
early  1977  have  been  reported  in  this  article. 
The  new  DODD  5000. 1 and  the  draft  DODD 
5000.3  have  been  analyzed.  The  views  of  vari- 
ous Department  of  Defense  and  Air  Force 
elements  on  the  effectiveness  of  AFTEC  and 
the  AFTEC  view  of  itself  were  compared.  Fi- 
nally, the  policies  and  views  were  combined 
into  recommendations  for  the  Program  Man- 
ager to  follow  to  ensure  that  the  valid  opera- 
tional requirements  are  in  the  development  pro- 
gram. Each  program  is  different  and  a universal 
set  of  guidelines  cannot  be  developed  that  will 
apply  to  all.  But  the  Program  Manager  will 
consider  all  guidelines  and  tailor  the  tactics  to 
his  specific  situation.  In  that  way  the  Program 
Manager  can  be  as  assured  of  success  as  anyone 
can  in  the  changing  political  and  military  envi- 
ronment of  today’s  world. 
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USE  OF  TRAINING  EXERCISES  FOR 
TEST  AND  EVALUATION 


by 

Lt  Col  Martell  D.  Fritz,  USA 
US  Army  Operational  Test  and  Evaluation  Agency 

How  many  times  have  you  beard  a major  automaker  or  other  industrial  giant  announce  the 
recall  of  an  expensive  product  for  correction  of  an  operational  or  safety  defect?  This  process  is 
costly,  time  consuming  and  can  severely  impact  the  future  growth  of  the  corporation  by  virtue 
of  consumer  dissatisfaction.  To  reduce  the  number  of  recalls,  industry  devotes  significant 
resources  to  improving  its  capability  to  predict  product  performance.  This  philosophy  began  to 
pervade  tbe  Department  of  Defense  community  during  the  David  Packard  era  and  became 
known  as  the  “fly  before  buy”  approach.  In  this  article  the  author  describes  bow  operational 
testing  combined  with  training  serves  to  deter  the  fielding  of  items  that  lack  operational 
effectiveness. 


INTRODUCTION 

In  May  1970,  the  Department  of  Defense  The  essence  of  Operational  Test  and  Eva/-  ‘ 

(DOD)  formally  recognized  the  need  for  inde-  nation  is  to  determine  the  usefulness  of  a system 

pendent  operational  test  and  evaluation  in  combat.  The  objective  is  to  measure  and  eval- 

(OT&E)  to  determine  system  operational  effec-  uate  a system’s  operational  effectiveness  and 

tiveness.  The  “fly  before  buy”  philosophy  was  suitability  based  upon  realistic  operational  con-  j 

to  be  fully  implemented  by  all  services.  Each  ditions.  The  distinction  betw'een  testing  and  j 

DOD  component  was  directed  to  establish  a evaluation  should  be  noted.  Operational  testing  j 

major  field  agency  and  give  it  responsibility  for  (OT)  is  the  gathering  of  information.  Evalua-  ■ 

the  Operational  Test  and  Evaluation  of  materiel  tion  involves  the  assessment  of  the  information  ] 

systems.  The  agency  was  directed  to  be  separate  gathered  to  achieve  an  objective  evaluation.  ’ 

and  distinct  from  the  development,  procure-  i 

ment  and  user  commands.  Further  the  Army  To  accomplish  its  mission,  OTEA  nor-  j 

determined  that  the  DOD  directive  could  best  mally  conducts  two  formal  sequential  tests  \ 

be  implemented  by  having  the  Army’s  indepen-  which  are  keyed  to  DOD  decision  reviews, 

dent  agency  report  directly  to  the  Army  Chief  Operational  Test  I (OT  1)  occurs  early  in  the  j 

of  Staff.  In  September  1972,  Department  of  the  materiel  acquisition  process.  Here  concentra-  ^ 

Army  (DA)  established  the  US  Army  Opera-  tion  is  on  the  primary  system  functions,  man-  i 

tional  Test  and  Evaluation  Agency  (OTEA)  machine  interfaces  at  the  operator  level,  and  the 

and  charged  this  Agency  with  the  Operational  demonstration  of  system  feasibility.  Opera- 

Test  and  Evaluation  mission  for  DA.‘  This  tional  Test  II  (OT  II)  occurs  following  full-  , 

Agency  would  provide  the  decisionmakers  with  scale  development  and  normally  is  the  most 

valid  test  data  and  independent  evaluations  of  rigorous  test  in  terms  of  obtaining  data  with  ’ 

system  suitability  in  keeping  with  the  threat,  statistical  confidence,  particularly  in  the  areas  < 

organizational  and  doctrinal  needs,  tactics,  of  functional  performance  and  reliability,  avail-  j 

techniques  and  training.  The  concept  was  fur-  ability  and  maintainability  (RAM),  The  Opera-  I 

ther  keyed  to  conduct  of  testing  with  typical  tional  Test  II  focuses  on  mission  requirements  ; 

soldiers  in  a realistic  operational  environment.  with  regular  troop  units  in  controlled  field  1 
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exercises  and  addresses  logistics  support  to  the 
maximum  extent.  The  results  of  OT  II  may  be 
used  to  support  a production  decision.  If  OT  II 
reveals  significant  deficiencies,  or  if  the  decision 
review  determines  that  critical  issues  remain 
unanswered,  the  decision  authority  may  direct 
that  corrections  be  made  and  that  appropriate 
check  tests  be  conducted  during  a contingency 
operational  test,  OT  Ila.  When  directed,  OTEA 
will  conduct  a follow-on  evaluation  subsequent 
to  the  production  decision  to  provide  informa- 
tion not  gained  from  previous  operational  test 
and  evaluation.  Additionally,  OTEA  approves 
the  plan  and  monitors  the  conduct  of  force 
development  testing  and  experimentation 
(FDTE).  The  FDTE  is  used  to: 

• develop  concepts  of  employment, 

• determine  operational  feasibility, 

• estimate  the  operational  advantages  of 
a proposed  system,  and 

• assist  in  the  development  of  require- 
ments documentation. 

The  Operational  Test  and  Evaluation 
Agency  is  the  Army  point  of  contact  with  the 
Office  of  the  Secretary  of  Defense  (OSD)  on 
joint  user  test  matters.  The  Agency  has  overall 
Army  management  responsibility  for  OSD- 
directed  joint  user  test  programs.  Joint  user 
tests  are  those  in  which  the  Army  participates 
with  one  or  more  of  the  other  Services  to 
evaluate  systems  or  concepts  having  an  inter- 
face with  or  requiring  a test  environment  of 
another  Service. 

REQUIREMENTS  FOR 
OPERATIONAL  TESTING 

Test  site  selection  is  influenced  by  many 
factors  Perhaps  the  most  important  factors  are 
test  environment,  availability  of  troops,  and 
host  installation  logistics  support.  The  test  envi- 
ronment includes  the  weather,  type  and  size  of 
terrain,  and  the  type  and  amount  of  vegetation. 
To  insure  validity  of  operational  testing  the  test 
environment  must  closely  approximate  the  ac- 
tual combat  conditions  under  which  the  system 


is  designed  to  function.  Equally  important  is 
the  availability  of  player,  aggressor  and  support 
troops.  The  troops  must  possess  the  required 
military  occupational  specialities  to  operate, 
maintain  and  support  the  system  under  test. 
Simulation  of  a realistic  operational  environ- 
ment often  necessitates  the  use  of  combat,  com- 
bat support,  and  combat  service  support  troops. 
If  these  troops  are  not  available  at  the  test 
installation,  such  troops  must  be  brought  in 
from  other  installations.  A significant  increase 
in  cost  is  a result.  A third  major  factor  influenc- 
ing test  site  selection  is  adequacy  of  logistics 
support.  For  example,  if  mechanized  infantry 
or  armor  units  are  involved,  a site  where  such 
units  are  assigned  is  preferred.  Due  consider- 
ation must  be  given  to  the  adequacy  of  adminis- 
tration and  supply  support  (billeting,  messing, 
office  space,  communications,  transportation, 
etc.)  required  for  test  directorate  personnel. 

After  the  location  has  been  selected  the 
Operational  Test  and  Evaluation  Agency  nor- 
mally will  send  a small  task  force  (three  or  four 
persons)  to  the  installation  to  discuss  test  re- 
quirements with  the  installation  commander 
and  his  staff.  The  primary  purpose  of  this  visit 
is  to  integrate  the  test  requirements  with  the 
training  objectives  of  the  units  selected  to  sup- 
port the  test.  The  coordination  visit  allows  the 
host  installation  to  participate  in  the  planning 
phase  and  is  a key  step  in  the  detailed  test 
planning  process. 

Since  testing  can  be  costly,  the  US  Army 
Operational  Test  and  Evaluation  Agency  is 
continually  searching  for  ways  to  reduce  costs 
without  degrading  the  adequacy,  quality  and 
credibility  of  tests.  Continual  coordination  with 
the  Program  or  Project  Manager,  the  develop- 
mental test  community,  and  the  host  installa- 
tion, on  cost  matters  provides  information  nec- 
essary to  address  cost  implications. 

USE  OF  TRAINING  EXERCISES 

The  combining  of  operational  testing  and 
training  exercises  can  benefit  the  attainment  of 
both  operational  test  and  training  objectives. 
One  way  to  attain  mutual  benefit  is  to  satellite 


an  operational  test  on  a planned  training  exer- 
cise. Another  valid  approach  is  to  define  the 
operational  test  objectives  and  requirements 
and  then  plan  a training  exercise  that  accom- 
modates those  requirements.  In  either  ap- 
proach, the  operational  test  objectives  are  met 
while  the  test  units  are  achieving  training 
objectives. 

The  Operational  Test  and  Evaluation 
Agency  has  gained  significant  experience  in  the 
use  of  training  exercises  to  accomplish  required 
operational  testing  and  evaluation.  This  experi- 
ence has  served  to  highlight  numerous  advan- 
tages and  to  identify  the  problem  areas  asso- 
ciated with  the  integration  of  operational  test 
and  evaluation  with  training  exercises.  The  fol- 
lowing examples  illustrate  how  training  exer- 
cises have  been  used  for  operational  test  and 
evaluation  and  the  relative  success  achieved. 

BART 

The  Baseline  Armor  Reliability  Test 
(BART)  was  conducted  to  determine  the  relia- 
bility of  the  new  M60A1,  the  overhauled 
M60A1,  and  the  overhauled/converted  M48A5 
tanks.  Five  tanks  of  each  type  were  driven  2250 
miles.  Each  tank  fired  450  rounds  from  its  main 
gun.  Data  concerning  repair  parts  usage,  bore- 
sight  zero  retention  of  the  main  gun,  and  sub- 
jective information  on  overall  tank  perform- 
ance were  obtained.  The  test  was  conducted  by 
the  TRADOC  Combined  Arms  Test  Activity, 
with  OTEA  participation,  at  Fort  Hood,  Texas 
(26  April  to  12  November  1976).  The  test  sup- 
port unit  was  the  2nd  Battalion,  8th  Armored 
Cavalry,  1st  Cavalry  Division.  The  test  in- 
volved three  phases  with  each  phase  consisting 
of  five  10-day  increments.  Test  events  were 
scheduled  and  controlled  by  the  test  unit  (con- 
sistent with  the  requirement  to  travel  150  miles 
and  fire  30  main  gun  rounds).  The  battalion 
commander  had  virtually  a free  hand  to  plan 
and  conduct  each  exercise.  The  battalion  was 
able  to  conduct  6 months  of  excellent  Army 
Training  and  Evaluation  Program  (ARTEP) 
training  without  cost  to  the  unit.  All  expenses 
were  paid  from  test  funds.  The  three  tank  com- 
panies completed  the  standard  tank  gunnery 
qualification  courses  and  conducted  extensive 
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platoon  level  training.  The  results  of  this  test  il- 
lustrate an  outstanding  example  of  the  in- 
creased combat  readiness  that  can  be  attained 
through  the  combination  of  testing  and  training 
exercises. 

DRAGON  OT  Ilia 


The  DRAGON  weapon  system  is  a com- 
mand linked,  line-of-sight  guided  missile  sys- 
tem. This  system  provides  firepower  for  the 
soldier  against  tanks,  armored  vehicles  and  for- 
tified battlefield  emplacements.  The 
DRAGON  OT  Ilia  was  conducted  at  Fort 
Bragg,  North  Carolina  (18  September  through 
10  December  1975).  The  primary  purpose  of 
the  test  was  to  determine  whether  the  approved 
logistical  support  concept  was  capable  of  sup- 
porting the  DRAGON  missile  system  in  addi- 
tion to  the  TOW  and  SHILLELAGH  systems 
(under  wartime  conditions)  and  to  assess  the 
adequacy  of  reliability  design  changes  to  the 
DRAGON  tracker.  In  Phase  III  of  the  test, 
three  maneuver  battalions  of  the  3d  Brigade, 
82d  Airborne  Division,  conducted  separate  10- 
day  tactical  field  exercises.  Each  battalion  de- 
veloped an  operations  order  that  incorporated 
the  tactical  scenario  for  the  test  as  well  as  the 
unique  trainingobjectives  of  the  battalion.  Each 
exercise  included  day  and  night  live  fire  exer- 
cises and  nonfiring  offensive,  defensive  and  re- 
trograde operations  against  a mobile  agressor 
force.  Live  fire  support  was  provided  by  Army 
aviation,  division  artillery,  and  Air  Force  tacti- 
cal aircraft.  Two  hundred  DRAGON  training 
missiles  and  nineteen  TOW  missiles  were  fired 
during  Phase  III.  This  exercise  proved  to  be 
highly  successful  in  terms  of  combining  an  op- 
erational test  with  a training  exercise.  The  bat- 
talions received  excellent  training  that  included 
the  benefits  of  the  additional  ammunition  allo- 
cated for  the  test.  Seventy-six  DRAGON  gun- 
ners were  rotated  through  the  live  fire  portion 
of  the  exercise,  enhancing  the  antiarmor  train- 
ing readiness  within  the  division.  Since  the  initi- 
al planning  of  the  battalion  commanders  incor- 
porated all  requirements  for  the  operational 
test,  the  training  exercises  were  conducted 
without  interference.  The  only  significant  test 
limitation  was  the  short  duration  of  the  exer- 
cises (10  days).  This  test  period  did  not  allow 
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sufficient  time  for  all  supply  actions  (parts  iden- 
tification, requisitioning,  processing,  delivery 
and  application)  to  occur. 

SOTAS  OT  I 


The  Stand-Off  Target  Acquisition  System 
(SOTAS)  consists  of  an  airborne,  long-range, 
high  resolution  Moving  Target  Indicator 
(MTI)  radar  and  a data  processing  and  display 
ground  station.  An  integrated  tracker/data  link 
connects  the  airborne  platform  to  the  ground 
station.  The  SOTAS  is  being  developed  to  pro- 
vide the  commander  with  an  improved  capabil- 
ity to  monitor  the  battlefield  and  provide  a 
target  acquisition  capability  for  engagement  of 
targets  at  long  ranges.'  The  SOTAS  underwent 
user  testing  in  September  1 976  during  the  exe- 
cution of  REFORGER  76  conducted  in  the 
Federal  Republic  of  Germany.  This  exercise 
was  a two-sided,  force-on-force,  free  play  tacti- 
cal exercise  conducted  by  units  of  the  North 
Atlantic  Treaty  Organization.  A UH-IH  heli- 
copter was  used  as  the  airborne  platform  for 
SOTAS.  The  Test  Design  Plan  for  the  SOTAS 
REFORGER  test  was  prepared  by  the  US 
Army  Combat  Developments  Experimentation 
Command  (CDEC).  In  the  planning  phase, 
OTEA  worked  extensively  with  CDEC  to  de- 
velop data  requirements  for  the  user  test — data 
requirements  that  would  be  comparable  to  the 
requirements  for  an  Operational  Test  I.  After 
the  user  test,  OTEA  determined  that  the  results 
were  adequate  to  waive  the  requirement  for  an 
Operational  Test  I of  SOTAS.  Although  the 
test  was  successful  from  an  OT  I standpoint 
(feasibility  proved  and  military  potential 
demonstrated),  some  limitations  were  evident. 
The  nature  of  the  free  play  exercise  made  it 
difficult  to  control  events  and  gather  precise 
data  such  as  the  true  location  of  targets  identi- 
fied by  SOTAS.  Additionally,  since  the  primary 
objectives  of  REFORGER  were  related  to 
training  and  readiness  with  the  SOTAS  test 
being  secondary,  the  troops  did  not  recognize 
the  role  of  test  team  data  collectors  and  did  not 
allow  data  collectors  access  to  the  operations 
center.  Thus,  the  flow  of  information  from 
SOTAS  to  the  Division  G2  could  not  be 
recorded  with  accuracy. 


MCTNS  DTII/OCT 

The  man-portable,  common  thermal  night 
sights  (MCTNS)  are  a family  of  night  sights 
developed  to  provide  a lightweight  long-range 
nighttime  capability  for  the  TOW  and 
DRAGON  missiles.  The  MCTNS  include  the 
Night  Observation  Device-Long  Range  (NOD- 
LR)  and  the  Ground/Vehicular  Laser  Locator 
Designator  Thermal  Night  Sight.  The  MCTNS 
are  designed  to  be  used  for  battlefield  surveil- 
lance, target  acquisition  and  missile  firing.  The 
Developmental  Test  Il/Operational  Climatic 
Test  of  MCTNS  in  a cold  region,  winter  envi- 
ronment was  conducted  near  Fort  Greely, 
Alaska  (January  through  April  1977).  A por- 
tion of  the  test  was  conducted  in  conjunction 
with  a battalion  level  Army  Training  and  Eval- 
uation Program  (ARTEP).  The  ARTEP  is  a 
diagnostic  tool  used  to  evaluate  performance 
and  program  training  to  achieve  a specified 
level  of  proficiency.'  Normally  a formal  battal- 
ion Army  Training  and  Evaluation  Program  is 
conducted  once  every  12  to  18  months  and 
often  serves  as  a basis  for  evaluating  the  perfor- 
mance of  the  battalion  commander.  During  the 
ARTEP  for  the  4th  battalion,  9th  Infantry,  two 
TOW  missile  squads  with  night  sights  and  one 
ground  surveillance  section  with  two  night  ob- 
servation devices-long  range,  were  attached  to 
the  battalion.  Numerous  benefits  accrued  from 
combining  the  MCTNS  test  with  the  Army 
Training  and  Evaluation  Program.  The  benefits 
included: 

• a reduction  in  resources  required, 

• a reduction  in  cost, 

• a cost  saving,  realistic  RAM  environ- 
ment, 

• a realistic  logistics  burden  for  the  test 
and, 

• use  of  real  world  tactics  and  doctrine. 

These  advantages  provide  strong  rationale  for 
extensive  use  of  ARTEP  for  accomplishing 
testing  requirements;  however,  there  are  some 
limitations  that  must  be  considered.  One  of 
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these  limitations  is  that  the  nature  of  the  Army 
Training  and  Evaluation  Program  makes  it 
difficult  to  replicate  events.  Although  replica- 
tion is  not  impossible,  such  as  in  a force-on- 
force  free  play  exercise,  any  test  that  requires 
considerable  replication  of  events  (for  statistical 
confidence)  would  tend  to  degrade  the  realism 
of  the  ARTEP.  A more  serious  limitation  is  the 
added  burden  that  is  placed  on  the  commander. 
Since  the  ARTEP  is  instrumental  in  evaluating 
his  performance,  a commander  is  not  usually 
receptive  to  additional  tasks  that  may  compli- 
cate mission  performance  during  the  ARTEP. 
The  commander  must  be  assured  that  the  data 
collected  in  the  test  will  not  be  used  for  any 
portion  of  the  ARTEP  evaluation.  Although 
the  MCTNS  Operational  Climatic  Test  did  not 
interfere  with  the  ARTEP  in  this  instance, 
many  tests  could  have  an  impact  on  the  train- 
ing, employment  and  logistics  missions  of  the 
battalion.  Thus,  it  is  apparent  that  each  test  has 
its  own  unique  requirements  and  not  all  sys- 
tems should  be  tested  in  conjunction  with  an 
Army  Training  and  Evaluation  Program. 

CORPS  AUTOMATION 
REQUIREMENTS  (CAR)— FORCE 
DEVELOPMENT  TESTING  AND 
EXPERIMENTATION  (FDTE) 

Project  Corps  Automation  Requirements 
is  an  evolutionary  program  designed  to  identify, 
validate  and  test  corps  management  informa- 
tion system  automated  data  processing  require- 
ments, less  Army  Tactical  Data  Systems,  for 
the  midrange  (3  to  8 year)  time  frame.’  In  the 
period  September  1976  to  March  1977,  the 
XVIII  Airborne  Corps  conducted  the  Project 
CAR  Baseline  Identification  Test  to  define  the 
current  Corps  automated  data  processing  base- 
line for  combat  service  support.  Subtest  IV  of 
this  force  development  and  experimentation 
was  conducted  in  conjunction  with  an  XVIII 
Airborne  Corps  Command  Post  Exercise, 
CABER  WARRIOR  IV,  at  Fort  Bragg,  North 
Carolina  (1  through  7 March  1977).  In  this  sub- 
test, the  14th  Data  Processing  Unit,  1st  Corps 
Support  Command,  operated  the  mobile 
S/360-40  computer  configuration  in  a field  en- 
vironment. The  purpose  of  this  subtest  was  to 


examine  the  test  unit’s  Modified  Table  of  Or- 
ganization and  Equipment  and  to  observe  oper- 
ating procedures  in  a tactical  environment.  Ad- 
ditionally, the  subtest  was  to  verify  certain  tac- 
tical information  requirements  identified  in 
earlier  subtests.  This  test  is  an  excellent  exam- 
ple of  employing  Operational  Test  and  Evalua- 
tion as  a satellite  on  a previously  scheduled 
training  exercise.  Both  the  exercise  and  the 
force  development  testing  and  experimentation 
were  successfully  completed  without  degrada- 
tion to  either. 

CEWI  BATTALION-FDTE 

The  Combat  Electronic  Warfare  Intelli- 
gence (CEWI)  Battalion  (Division)  is  a concep- 
tual organization  that  consolidates  all  tactical 
electronic  warfare  and  intelligence  resources 
(personnel  and  equipment)  to  be  assigned  to  a 
US  Army  division.  The  battalion  organization 
is  being  developed  in  response  to  a recommen- 
dation made  in  a Department  of  Army  Intelli- 
gence Organization  and  Stationing  Study,  a 
study  in  which  tactical  intelligence  integration 
was  addressed.  The  study  group  recommended 
the  placing  of  tactical  support  resources  in  sin- 
gle units  under  the  command  and  control  of 
tactical  Army  commanders.  Because  of  this  re- 
commendation, the  CEWI  Battalion  was  evalu- 
ated by  the  TRADOC  Combined  Arms  Test 
Activity  (TCATA)  during  GALLANT  CREW 
77  (a  Joint  Chiefs  of  Staff  directed  US  Readi- 
ness Command  joint  readiness  exercise). 
GALLANT  CREW  77  was  conducted  at  Fort 
Hood  and  other  locations  in  central  Texas  (25 
to  31  March  1977).  The  purpose  of  this  force 
development  test  and  experimentation  was  to 
evaluate  the  concept  of  organization  and  opera- 
tion for  the  CEWI  Battalion  of  the  2nd  Ar- 
mored Division.  Although  the  objectives  of  the 
exercise  were  met  in  an  outstanding  manner  ex- 
cept for  the  conduct  of  operations  in  an  elec- 
tronic warfare  environment,’  the  force  develop- 
ment testing  and  experimentation  were  sub- 
stantially less  successful.  There  were  three  ma- 
jor reasons  for  the  limited  success  of  the  force 
development  testing  and  experimentation. 
Probably  most  important  was  the  lack  of  early 
involvement  by  the  Operational  Test  and  Eval- 
uation Agency  in  the  planning  phase.  The  late 
entry  of  OTEA  limited  the  ability  of  the  test 
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team  to  interject  situations  into  the  exercise  • /fV/CFVOr//.TheInfantryFight- 

that  would  stress  functional  areas  of  the  CEW!  ing  Vehicle/Cavalry  Fighting  Vehicle  (for- 

Battalion,  Because  of  the  free  play  nature  of  the  merly  known  as  the  MICV)  OT  II  is  scheduled 

exercise,  the  US  Readiness  Command  was  re-  for  1979.  The  Infantry  Fighting  Vehicle  is  the 

luctant  to  interject  any  artificiality  into  the  see-  companion  vehicle  for  the  XM 1 Tank  and  must 

nario  for  fear  of  jeopardizing  attainment  of  the  be  tested  in  a combined  arms  operational  envi- 

exercise  objectives.  Secondly,  severe  personnel  ronment.  Although  the  test  does  not  appear  to 

and  equipment  problems  affected  the  sensitivity  be  appropriate  for  use  as  a satellite  on  a training 

ofthe  test  data.  There  was  a shortage  of  person-  exercise,  it  should  be  extensive.  This  test  has 

nel  having  certain  skilled  MOS  and  personnel  great  potential  for  thorough  combined  arms 

with  required  security  clearances.  The  equip-  training  of  the  test  units  with  a resultant  im- 

ment  shortages  occurred  because  of  low  item  provement  in  combat  readiness, 

density  in  special  types  of  electronic  warfare 

equipment.  Thirdly,  the  level  of  training  of  the  • EW/CAS  Joint  Test,  The  Electronic 

Battalion  was  less  than  desired.  The  training  Warfare  During  Close  Air  Support  (EW/CAS) 

shortfall  resulted  from  limited  training  time  Joint  Test  is  a US  Army/US  Air  Force/US 

and  late  key  personnel  and  equipment  fills.  Marine  Corps  operational  test  to  be  conducted 

during  the  period  1978  through  1980.  The  pur- 
FUTURE  POSSIBILITIES  pose  is  to  determine  the  impact  of  an  intense 

electronic  warfare  environment  on  command. 

The  success  achieved  to  date  provides  a control  and  communications  systems.  Empha- 

strong  incentive  to  examine  future  testing  for  sis  will  be  on  close  air  support  and  attack 

possible  integration  with  training  exercises.  helicopter  support  during  combined  opera- 

There  are  numerous  tests  that  may  fall  into  this  tions.  The  scope  and  nature  of  this  test  offer  an 

category.  Some  ofthe  more  promising  possibili-  opportunity  to  combine  testing  with  division, 

ties  are  outlined  here.  brigade,  or  battalion  training,  or  joint  exercises. 

• Division  Restructuring  Study  BENEFITS 
FDTE.  The  purpose  of  this  force  development 

testing  and  experimentation  is  to  evaluate  the  The  .Army’s  efforts  to  combine  testing 

overall  effectiveness  of  a restructured  heavy  di-  with  training  exercises  have  yielded  tangible 

vision.  Phase  II  of  the  test  currently  is  sched-  benefits.  The  results,  although  some  are  diffi- 

uled  to  be  conducted  during  1978.  Envisioned  is  cult  to  quantify,  definitely  contribute  to  the 

the  testing  of  a division  (minus)  in  an  operation-  overall  goal  of  improved  combat  readiness.  Cer- 

al  environment.  The  test  will  involve  elements  tain  significant  benefits  are  summarized  below, 

of  III  Corps,  1st  Cavalry  Division,  and  the  2nd 

Armored  Division.  This  phase  of  the  test  offers  • Improved  command  readiness. 

excellent  training  opportunities  and  is  a candi-  Because  of  the  extra  training  received  in  con- 

date  for  integration  with  a large  exercise.  junction  with  an  operational  test  many  test 

units  have  been  able  to  achieve  a much  higher 

• M198  Follow-on  Evaluation.  The  level  of  combat  readiness  than  had  been  previ- 

M198  155mm  towed  medium  howitzer  was  ously  obtained.  Commanders  at  all  levels  are 

type  classified  and  approved  for  production  on  concerned  about  the  cost  of  training  and  the 

14  October  1976.  The  purpose  ofthe  follow-on  attainment  of  training  objectives  within  budget 

evaluation,  currently  scheduled  for  Oct  78-Jan  limitations.  Budget  constraints  usually  result  in 

79,  is  to  verify  that  system  deficiencies  have  the  establishment  of  definite  limits  on  the  quan- 

been  corrected  in  the  areas  of  reliability,  avail-  tity  of  fuel,  ammunition  and  spare  parts  avail- 
ability and  maintainability,  tactical  mobility,  able  to  support  training  objectives.  The  BART 

training,  and  human  factors  and  in  the  concept  and  DRAGON  OT  Ilia  are  two  excellent  ex- 

of  logistic  support.  The  test  appears  ideal  for  amples  where  commanders  improved  unit  corn- 

integration  with  pre-ARTEP  training  of  the  bat  readiness  by  innovative  and  resourceful 

initial  operational  capability  unit.  planning  for  tests. 
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• Realistic  test  environment.  Train- 
ing exercises  that  are  planned  and  executed  by 
combat  units  usually  provide  a more  realistic 
environment  for  a test  than  do  exercises  that  are 
planned  by  “expert  testers"  using  “school  solu- 
tion" doctrine  and  tactics.  Real  world  problems 
and  limitations  are  more  likely  to  appear  in 
training  exercises  than  in  "t  inned”  scenarios. 

• Cost  savings.  Cost  it  an  important 
consideration  in  every  test.  Cost  often  becomes 
a critical  factor  in  size  and  scope  of  test 
determinations.  As  systems  become  more  com- 
plex and  testing  becomes  more  expensive,  every 
possible  cost  reduction  must  be  explored.  Ac- 
cordingly, placing  satellite  tests  on  previously 
scheduled  training  exercises  may  become  more 
of  the  norm  in  the  future. 

• Realistic  logistics  burden  (in- 
cluding reliability,  availability  and 
maintainability).  A difficult  task  in  the  de- 
sign of  a test  is  the  creation  of  an  environment 
that  will  provide  a reliable  indication  of  the 
logistics  supportability  of  a system  and  the 
adequacy  of  reliability,  availability  and  main- 
tainability characteristics.  Usually  the  opera- 
tional environment  is  simulated  by  employing  a 
small  slice  of  the  direct  support  and  general 
support  capability  and  dedicating  this  small 
contingent  to  the  support  of  the  system  being 
tested.  Thus  the  support  units  are  rarely  placed 
in  a “true”  operational  environment.  One  obvi- 
ous advantage  of  testing  in  conjunction  with 
large  training  exercises  is  that  support  units 
must  function  within  an  overall  tactical  context 
and  provide  support  for  the  various  types  of 
units  and  equipment  used  in  the  exercise.  Any 
additional  logistic  burden  imposed  by  the  sys- 
tem being  tested  will  be  readily  identified. 

• Reduced  test  support  require- 
ments. Support  of  user  testing  involves  exten- 
sive planning  to  provide  for  administrative  and 
logistic  support  of  test  directorate  and  player 
personnel,  use  of  aggressor  forces,  control  of 
personnel,  training,  availability  of  equipment 
and  numerous  minor  tasks.  Many  support  re- 
quirements may  be  significantly  reduced  when 
testing  is  integrated  with  a training  exercise. 


• Development  of  doctrine  and 
tactics.  Training  exercises  better  accommo- 
date the  evaluation  and  refinement  of  doctrine 
and  tactics  for  the  system  being  tested.  In  the 
context  of  a training  exercise  real  world  prob- 
lems can  be  surfaced  and  possibly  resolved 
prior  to  the  fielding  of  a system. 

PROBLEM  AREAS 

The  Army’s  experience  with  operational 
test  and  evaluation  has  revealed  problem  areas 
that,  in  some  cases,  limit  the  value  of  combining 
testing  with  training  exercises.  These  limita- 
tions underscore  the  need  for  caution  before  a 
decision  is  made  to  combine  a test  with  a 
training  exercise.  Significant  problem  areas 
include; 

• Inability  to  control  events.  It  is  im- 
practicable to  control  events  in  a free 
play  exercise.  Although  this  constraint 
may  not  affect  some  tests,  in  other  tests 
it  is  essential  that  certain  system  capa- 
bilities be  tested  or  stressed.  When  con- 
sidering a free  play  exercise,  the  system 
must  be  reviewed  to  insure  that  critical 
capabilities  will  be  completely  tested.  If 
this  assurance  cannot  be  obtained,  the 
exercise  will  be  of  little  value  in  meet- 
ing the  test  objectives.  This  fact  was 
amply  demonstrated  during  the  CEWI 
Battalion  force  development  test  and 
experimentation. 

• Possible  burden  on  tbe  Commander 
during  tbe  ARTEP.  A formal  Army 
Training  and  Evaluation  Program  is  a 
key  period  of  evaluation  for  a com- 
mander. Any  test  that  places  an  addi- 
tional burden  on  the  unit  taking  an 
ARTEP  probably  would  not  be  well 
received  by  the  commander.  Not  only 
would  the  combined  test  be  unfair  to 
the  commander  but  also  the  system 
being  tested  probably  would  not  be 
employed  to  maximum  capability.  This 
situation  does  not  preclude  the  possi- 
bility of  using  an  Army  Training  and 
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Evaluation  Program  as  a test  vehicle, 
but  it  does  severely  limit  the  instances 
where  use  of  such  vehicle  would  be 
feasible.  One  type  of  system  that  could 
be  tested  is  one  that  provides  a comple- 
mentary increase  in  capability  as  op- 
posed to  a complete  replacement  sys- 
tem requiring  extensive  reorganization 
and/or  training. 

• Possible  interference  with  control  and 
data  collection.  Every  test  has  a re- 
quirement for  a certain  number  of  con- 
trollers, data  collectors  and  deputy  test 
directors.  These  personnel  must  be  able 
to  observe  and  record  critical  activities 
during  the  tests.  This  requirement  usu- 
ally means  that  these  persons  must 
have  access  to  operations  centers,  com- 
mand posts,  fire  direction  centers  and 
other  key  locations.  Accommodation 
of  test  personnel  can  become  a problem 
and  can  place  unnecessary  pressure  on 
the  unit  being  evaluated  if  the  activities 
are  not  carefully  planned  and  con- 
trolled. Herein  lies  the  real  challenge  to 
the  operational  tester.  Test  design 
plans  must  be  thorough  and  efficient. 
The  methodology  must  be  used  to  opti- 
mize test  data  without  hampering  any 
portion  of  the  Army  Training  and 
Evaluation  Program.  Equally  impor- 
tant is  the  perception  by  members  of 
the  tested  unit  that  the  data  collected 
from  the  test  will  affect  the  evaluation 
during  the  Program.  The  test  director 
can  play  an  important  role  in  preclud- 
ing such  apprehensions  by  ensuring 
that  test  planning  specifically  calls  for 
careful  separation  of  test  data  from 
that  required  for  the  Army  Test  and 
Evaluation  Program.  Such  test  control 
matters  should  be  highlighted  during 
pretest  coordination  meetings. 

• Insufficient  replication  of  events.  Rep- 
lication, in  a scientific  sense,  is  the 
systematic  repetition  of  a test  event 
under  identical  conditions  based  on  the 
approved  test  design  plan.  Replication 
often  is  required  to  reduce  the  chance 


of  errors  entering  into  the  collection  of 
test  samples.  In  a training  exercise  any 
significant  amount  of  replication  be- 
comes difficult  as  the  tactical  realism 
of  the  exercise  tends  to  diminish.  A 
scenario  driven  exercise  or  the  use  of 
side  excursions  (tactical  snapshots)  can 
often  provide  needed  replication  of 
events. 

SUMMARY 

The  Army’s  own  study  of  the  materiel 
acquisition  process  supported  the  need  for  a 
separate  and  independent  Operational  Test  and 
Evaluation  Agency.  The  Army  recognized  that 
because  of  the  increasing  expense  and  complex- 
ity of  materiel  systems  the  systems  must  be 
placed  in  the  hands  of  typical  user  troops  as 
early  as  practicable  in  the  development  process 
and  evaluated  prior  to  costly  production  deci- 
sions. The  challenge  facing  the  operational  tes- 
ter is  to  be  able  to  accomplish  that  mission  at 
minimum  cost. 

Guided  by  the  important  phrase  “In  the 
hands  of  typical  user  troops,”  one  attractive 
way  of  fulfulling  the  operational  tester’s  mis- 
sion might  be  an  increase  in  the  use  of  training 
exercises  to  accomplish  test  and  evaluation  ob- 
jectives. The  Army’s  operational  tester,  the  Op- 
erational Test  and  Evaluation  Agency,  contin- 
ues to  depend  upon  the  resources  of  Forces 
Command,  TRADOC,  and  other  organizations 
to  accomplish  operational  tests  of  major  and  se- 
lected nonmajor  (Category  I)  systems.  The  cost 
effectiveness  of  integrating  operational  testing 
with  training  exercises  is  evident.  However, 
other  operational  considerations  must  be  ex- 
amined to  insure  that  this  approach  meets  ap- 
proved test  objectives.  The  advantages  to  be 
gained  from  the  integration  of  operational  test- 
ing with  training  exercises,  when  feasible,  far 
outweigh  problem  areas  encountered  to  date. 
The  bonus  effects  are  improved  training  oppor- 
tunities and  increased  combat  readiness.  Al- 
though training  exercises  cannot  be  viewed  as  a 
universal  solution,  these  exercises  do  offer  an 
attractive  option  for  conduct  of  operational  test 
and  evaluation. 


REFERENCES 


Cited  References 


1.  Dept  of  Army,  “Operational  Test  and  Evaluation  Methodology  Guide,”  US  Army  Operational  Test  and  Evalua- 

tion Agency,  May  76. 

2.  , “SOT AS  (REFORGER)  Test  Design  Plan,"  US  Army  Combat  Developments  Experimentation  Com- 
mand, Jul  76. 

3.  , “Army  Training  and  Evaluation  Program  for  General  Support  Cannon  Units,  Department  of  the 
Army,”  ARTEP  .No.  6-165,  Sep  76. 

4.  . , “Corps  Automation  Requirements  Baseline  Identification  Test,  Test  Design  Plan,”  FDTE  Project  No. 

FO-049,  Project  CAR  Test  HQ,  XVIII  Airborne  Corps;  HQ,  US  Army  Training  and  Doctrine  Command;  and 
HQ,  US  Army  Logistics  Center;  10  Sep  76. 

5.  , HQ,  III  Corps,  AFZF-DPT-PLX-E  to  CINC  US  Readiness  Command,  I.tr,  subject:  “Executive  Sum- 

mary, GALLANT  CREW  77  After  Action  Report,”  6 May  77. 


Additional  References 

Dept  of  Army,  OTEA,  CSTE-EDS,  Memorandum  for  Record,  subject:  “OTEA  Position  on  SOT AS  OT  I,”  30  Nov 
76. 

, “Army  Training  and  Evaluation  Program  for  Light  Infantry  Battalions,"  ARTEP  7-14, 30  Apr  76. 

“Project  CAR  Aids  Commanders,”  The  ParagHde,  31: 8 (1977)  Ft  Bragg,  NC,  10  Mar  77. 

Dept  of  Defense,  “Major  System  Acquisitions,”  Directive,  Number  5000.1, 18  Jan  77. 

, “Test  and  Evaluation,”  Directive,  Number  5000.3,  Ch  2, 20  May  77. 


Lt  Col  Marlell  D.  Fritz 
is  Assistant  to  the  Scientific 
Advisor,  US  Army  Opera- 
tional Test  and  Evaluation 
Agency,  Falls  Church,  VA. 

Lt  Col  Fritz  has  served  in 
command  of  Battery  B,  2d 
Howitzer  Battalion.  5th 
Fielo  Artillery,  USAREUR,  1964-65;  Battery  B,  2d  Bat- 
talion. 19th  Field  Artillery,  Ist  Cavalry  Division,  Viet 
Nam,  1967;  and,  as  Commander,  USA  NCO  Acade- 
my/Drill Sergeant  School,  Fort  Knox,  KY.  1974  He  has 
held  staff  assignments  at  the  Gunnery  Dept,  Fort  Knox. 


KV,  1967-69;  in  the  TACFIRE  Project  Managers  Office, 
Fort  Belvoir,  VA,  1969-71;  and  prior  to  his  present  as- 
signment, in  the  office  of  Jl,  United  Nations  Command, 
Korea,  1972-73. 

Lt  Col  Fritz,  a contributor  of  articles  to  "The  Field 
Artilleryman,"  has  completed  both  the  Field  Artillery  Of- 
ficers Basic  Course,  1962,  and  the  Artillery  Officers  Ad- 
vance Course,  1966,  at  Fort  Sill,  OK.  He  later  completed 
the  US  Army  Command  and  General  Staff  Course  at  Fort 
Leavenworth,  KS,  1972. 

Lt  Col  Fritz  received  his  BS.  Military  Science  from 
the  US  Military  Academy  in  1961  and  an  MS,  Systems 
Management,  University  of  Southern  California,  1975. 


118 


Defense  Systems  Management  Review 


CHALLENGES  IN  AIRCRAFT  ENGINE 
TEST  AND  EVALUATION  PROGRAMS 

by 

David  T.  Love,  Manager 

Engine  Evaluation  Engineering,  Aircraft  Engine  Group 
General  Electric  Company 


Many  important  factors  affect  an  aircraft  engine  test  and  evaluation  program,  whether  for 
military  or  commercial  application.  These  factors  in  turn  affect  the  engine  development  cycle 
requirements  and  the  overall  test  and  evaluation  process.  Often  new  challenges  are  presented 
and  sometimes  surprises  as  each  new  system  is  developed.  In  this  article  the  author  looks  at  a 
few  of  the  factors  and  notes  how  they  relate  to  the  development  cycle.  Some  areas  of 
improvement  are  suggested  and  the  basic  requirements  of  the  test  and  development  cycle  are 
given. 


THE  ENGINE  DEVELOPMENT  CYCLE 

A typical  cycle  in  the  development  of  ma- 
jor engine  components  into  an  advanced  state- 
of-the-art  engine  is  shown  in  Figure  1.  This  de- 
velopment cycle  is  typical  of  what  is  required  to 
assure  that  a new  engine  meets  design  require- 
ments. The  cycle  is  based  on  30  years  of  engine 
development  experienced  at  one  company.  As 
shown,  a period  of  basic  applied  research  and 
component  testing  precedes  full-scale  engine 
development  to  bring  forth  a state-of-the-art 
application  of  advanced  technology.  Then,  as 
engine  testing  is  initiated,  both  component  and 
full-scale  engine  testing  progress  toward  quali- 
fication (or  certification),  flight  test  and  eventu- 
ally into  production.  What  is  not  shown  are  the 
advanced  technology  Advanced  Turbine  En- 
gine Gas  Generator  (ATEGG),  Aircraft  Pro- 
pulsion System  Integration  (APSI)  type  dem- 
onstrator programs  that  precede  these  events. 
The  demonstrator  programs  provide  the  tech- 
nology base  for  the  new  systems  and  are  vital  to 
maintaining  technological  leadership.  In 
production,  service  life  verification  and  contin- 
ued improvement  activities  logically  follow. 
Few  shortcuts  have  been  found  that  can  com- 
press this  development  cycle.  Attempts  to  leave 
out  steps  in  the  evaluation  process  usually  mean 
a costly  return  to  do  an  omitted  task. 


Comparison  of  the  test  and  evaluation  cy- 
cles of  an  engine  for  a commercial  vs  a military 
application  shows  the  requirements  to  be  paral- 
lel. In  almost  every  case,  the  commercial  en- 
gines of  today  were  derived  from  engines  origi- 
nally developed  for  a military  application. 
Many  of  the  major  components — such  as  the 
complete  core  engine  system  of  the  TF39/CF6 
derivative  engines — were  first  developed  and 
qualified  under  a military  program.  As  a result, 
the  derivative  engines  can  be  certified  in  a 
shorter  time  period  when  a relatively  modest 
advancement  in  technology  is  accomplished. 
An  example  of  this  is  the  CF6-6  and  -50  com- 
mercial engines,  which  were  derived  from  the 
TF39  engine.  The  TF39  engine  was  an  out- 
growth itself  of  a military  sponsored  research 
and  development  program  (the  Advanced  Tur- 
bine Engine  Gas  Generator). 

As  shown  in  Figure  2,  the  engines  were  de- 
veloped and  certified  in  a relatively  short  time. 
Greater  emphasis  was  placed  on  developing  the 
improved  durability  and  maintainability  char- 
acteristics. These  enhanced  characteristics  were 
required  by  the  airlines  to  maintain  a competi- 
tive position  in  the  industry  relative  to  cost-of- 
ownership  and  utilization  rate  requirements. 
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Note  that  because  of  the  derivative  engine  rela- 
tionship most  of  the  durability  and  maintaina- 
bility improvements  were  incorporated  into  the 
TF39  engine. 

Preproduction  certification  test  require- 
ments are — and  have  been  for  some  time — 
getting  tougher.  Emphasis  has  been  shifting  to- 
ward a better  balance  among  performance,  ini- 
tial development  costs,  and  cost-of-ownership 
objectives.  The  producers  and  users  (companies 
and  customers),  both  military  and  commercial, 
are  getting  more  sophisticated.  The  demands 
for  better  simulation  and  more  accurate  repre- 
sentation of  operating  conditions  and  capabili- 
ties have  resulted  in  a complex  series  of  test 
demonstration  requirements.  The  reliability 
and  serviceability  of  engines  are  continually  im- 
proving and  each  of  the  new  development  pro- 
grams, in  one  specific  company,  has  been  struc- 
tured to  bring  the  engines  to  a development 
stage  of  greater  maturity.  These  actions  are 
taken  to  minimize  the  risks  and  potential  costs 
involved  in  a premature  production  commit- 
ment. This  translates  to  a more  complete  devel- 
opment program  and  also  to  increased  test  re- 
quirements and  increased  test  development 
program  costs. 


As  an  example.  General  Electric  recently 
completed  a very  extensive  and  complicated  se- 
ries of  engine  and  component  qualification  tests 
on  the  FlOl  engine,  which  is  being  developed 
for  the  B-1  strategic  bomber.  The  test  require- 
ments included  two  complete  back-to-back  157 
hour  extra  severity  endurance  cycle  tests  on  the 
same  engine,  with  power  usage  cycles  and  oper- 
ating conditions  tailored  to  the  B- 1 mission  pro- 
files. Three  hundred  and  seventy  hours  were  ac- 
cumulated on  this  engine  representing  more 
than  1000  hours  of  anticipated  service  opera- 
tions. Simultaneously,  another  engine — 
installed  in  the  engine  altitude  test  facility  at 
Arnold  Engineering  Development  Center — 
completed  a thorough  performance  calibration 
test  series,  inlet  distortion  tolerance  demonstra- 
tions, and  stall  margin  capability  demonstra- 
tions at  several  key  points  in  the  flight  envelope. 
These  two  particular  tests — which  represent 
only  part  of  the  overall  series  of  qualification 


test  demonstration  requirements — easily  sur- 
pass anything  that  has  been  accomplished  cn 
prior  test  programs. 

If  this  test  program  has  succeeded  in  pro- 
viding an  accurate  simulation  of  the  environ- 
mental operating  conditions  and  power  usage 
requirements  that  the  engine  will  be  exposed  to 
in  service,  then  initial  service  life  and  reliability 
objectives  will  have  been  achieved.  Achieve- 
ment of  these  objectives  will  in  turn  result  in  a 
smoother  and  more  trouble-free  flight  opera- 
tion, and  a significant  long  term  cost  savings  to 
the  program. 

There  are  a few  areas  where  some  mean- 
ingful improvements  could  be  made  in  the  test 
and  evaluation  process. 

NEW  APPROACHES  TO 
DEVELOPMENT  TESTING 

The  basic  challenge  is  clear.  Costs  are  in- 
creasing. Test  requirements  are  increasing  and 
test  costs  are  increasing.  We  must,  on  a contin- 
uing basis,  push  to  improve  our  state-of-the-art 
test  simulation  and  test  measurement  capabili- 
ties to  obtain  an  increased  quantity  of  meaning- 
ful information  from  each  test — more  informa- 
tion per  test  hour  and  more  information  per 
dollar.  New  techniques  both  in  data  acquisition 
and  reduction  are  available  and  must  be  imple- 
mented to  speed  and  improve  the  quality  and 
quantity  of  data  to  be  obtained  in  the  total  eval- 
uation process.  This  means  investment  in 
equipment,  research,  software,  and  new  ap- 
proaches to  development  testing. 

An  entirely  new  computer  data  system  has 
been  designed  to  improve  engineering  evalua- 
tion effectiveness.  This  data  system  will  have  a 
major  impact  on  the  way  engine  tests  are  con- 
ducted and  on  the  management  of  test  pro- 
grams. The  key  to  the  new  system  is  the  avail- 
ability of  an  interactive  graphic  display  that  the 
engineer  can  use  to  view  the  data  being  taken 
“on  line”  and  plotted  in  engineering  units. 
These  plots  will  be  compared  with  the  predicted 
results,  immediately,  in  a graphic  display  so 
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that  decisions  can  be  made  on  the  course  of  the 
test  program  and  the  success  or  failure  of  the 
test  being  conducted. 

This  new  system  will  impact  the  manage- 
ment of  the  test  and  evaluation  process  by  pro- 
viding a number  of  benefits  and  challenges. 
Some  of  the  benefits. 

• More  data  will  be  available  on  line  and 
plotted  for  faster  learning  and  decision 
making. 

• The  data  as  presented  graphically  can 
be  edited  so  that  only  data  of  interest  is 
analyzed. 

• A hard  copy  plot  of  the  data  will  be 
available  eliminating  the  time  and 
“dog  work”  of  plotting  data. 

• Test  time  can  be  reduced  by  a faster  as- 
sessment of  data  (elimination  of  unnec- 
essary test  points)  so  that  testing  can 
proceed. 

• An  increased  percentage  of  data  can  be 
analyzed. 

• Engineering  time  can  be  used  more  ef- 
fectively for  analysis  and  decisionmak- 
ing. 


Some  challenges  will  be; 

• A greater  demand  on  engineering  per- 
sonnel to  predict  test  results  for  on  line 
comparison  and  analysis. 

• A need  to  have  alternate  courses  of  ac- 
tion as  test  results  uncover  problem 
areas. 

• Test  and  evaluation  programs  will  de- 
mand more  forethought. 

• As  the  evaluation  process  is  speeded,  a 
demand  for  faster  problem  fixes  and 
test  turn  around  will  follow. 


Figure  3 shows  a typical  type  of  graphic 
plot  available  on-line  where  the  actual  test  data 
is  compared  with  the  engineering  prediction.  In 
addition,  an  on  line  calculated  projection  gives 
the  test  director  an  indication  of  what  may  lie 
ahead  as  the  test  proceeds. 

Many  improvements  in  engine  and  compo- 
nent test  facilities  are  being  made  to  keep 
abreast  of  the  advancements  in  engine  design. 
High  energy  X-ray  of  a running  engine,  with  vi- 
sual representation  of  mechanical  phenomena 
occurring  inside  the  engine,  is  being  used  to 
augment  analytical  techniques.  Some  recent 
work  with  laser  measurement  techniques  to  ob- 
tain jet  velocity  profiles  or,  in  one  case,  icing 
cloud  droplet  size  information,  appear  very 
promising.  Certainly,  one  of  the  key  elements  in 
better  test  and  evaluation  lies  in  the  improve- 
ment of  the  testing  facilities  both  in  government 
and  industry. 

ESTABLISHING  AND  INTERPRETING 
REQUIREMENTS 

A better  job  needs  to  be  done  of  establish- 
ing and  interpreting  basic  “design-to”  and 
“test-to”  certification  requirements.  Im- 
provement is  needed  in  the  ability  to  feedback 
and  interpret  information  gained  from  opera- 
tional experience.  Improvement  is  needed  in  the 
ability  to  translate  basic  cost  and  usage  objec- 
tives into  meaningful  test  requirements  tor  use 
in  providing  valid  technical  measure?  of  essen- 
tial capabilities.  It  is  to  be  recognized  that  accu- 
rate simulation  of  all  inservice  operating  condi- 
tions is  impossible.  Many  of  our  current  re- 
quirements are  based  on  subjective,  and  some- 
times inaccurate,  interpretation  of  “lessons 
learned  in  the  field.”  Also,  the  pass-fail  conno- 
tations associated  with  certification  test  pro- 
grams too  easily  can  be  interpreted  as  a guaran- 
tee that  unforeseen  problems  will  not  occur  in 
the  field  after  an  engine  has  been  certified. 

In  normal  service,  engines  ingest  tires, 
birds,  runway  debris,  etc.,  and  are  expected  to 
operate  with  minimum  damage  and  achieve  a 
high  standard  of  reliability.  Normal  factory 
testing  requires  limited  ingestion  testing  of 
small,  medium,  and  large  typical  objects  in 
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CORRECTED  SPEED  PERCENT 


Figure  3.  Demonstration  of  Predicted  Values  for  Bearing  Temperature  Together  with 
Measured  Data  (X's)  and  On  Line  Calculated  Projections  (O's) 


specified  quantities.  If  these  certification  tests 
are  passed,  the  engine  is  determined  to  be  satis- 
factory for  service. 

Experience  has  shown  that  ingestion  dam- 
age, particularly  that  caused  by  bird  strikes,  is 
quite  unpredictable.  One  ingestion  incident 
may  not  produce  damage  while  another  inges- 
tion incident  of  the  same  type  may  produce  sig- 
nificant damage.  Statistically,  a single  ingestion 
test  run  in  the  development  program  may  not 
produce  the  results  seen  in  operational  service 
and  thus  conclusions  as  to  the  satisfactory  oper- 
ation of  the  engine  during  ingestion  experiences 
may  not  be  correct. 


An  approach  in  trying  to  make  engine  de- 
velopment testing  more  compatible  with  field 
experience  is  to  produce  the  effects  of  major  for- 
eign object  ingestion,  that  may  include  high  im- 
balance owing  to  blade  breakage,  and  to  operate 
the  engine  with  predicted  levels  of  imbalance  to 
determine  the  effects  on  engine  operation.  This 
approach  removes  the  variability  of  certain  in- 
gestion tests  and  substitutes  the  damage  pre- 
dicted for  a worst  instance  of  field  ingestion.  It 
is  extremely  beneficial  to  determine  problems  of 
this  kind  during  development  testing  in  the  fac- 
tory rather  than  after  the  many  hours  of  service 
operation  anticipated  to  produce  such  ingestion 
caused  damage. 
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An  example  of  the  type  of  test  performed 
to  evaluate  the  effects  of  imbalance  due  to  for- 
eign object  damage  is  given  here.  Two  large  sec- 
tions of  two  fan  blades  were  intentionally  se- 
vered while  the  engine  was  operating  at  full 
takeoff  power  to  simu'ate  the  damage  experi- 
enced under  the  worst  conditions  in  the  field 
This  amount  of  imbalance  can  be  used  as  a 
“baseline”  for  the  evaluation  of  all  components 
within  the  system. 

PROGRAM  MANAGEMENT  AND 
THE  "RIGHT-TO-FAIL" 

Finally,  a few  words  about  development 
program  management  aspects.  A well  thought- 
out  and  properly  executed  test  and  evaluation 
program  plays  a significant  and  vital  role  in  the 
development  process  of  many  complex  systems. 
The  selection  and  use  of  certain  major  test 
events  as  program  progress  indicators  is  both 
natural  and  effective. 

The  development  of  a complex  system, 
such  as  an  advanced  propulsion  system,  is  typi- 
cally an  iterative  process.  Feedback  of  informa- 
tion gained  from  the  test  program  is  translated 
into  refined  “design-to”  requirements.  Also,  in 
some  cases,  this  feedback  information  is  trans- 
lated into  refined  “test-to”/”certify-to”  re- 
quirements as  in  the  object  ingestion  problem 
just  discussed. 

In  a typical  development  program  trade- 
offs are  necessary.  Unforeseen  problems  do  oc- 
cur, and  management  system  flexibility  is 
needed  to  keep  the  program  on  track  and  to 
maintain  a proper  balance  among  cost,  sched- 
ule, and  performance  objectives. 

The  selection  and  interpretation  of  individ- 
ual test  program  events  as  major  contract  mile- 
stones must  be  done  carefully  to  avoid  placing 
too  much  emphasis  on  the  "successful  comple- 
tion” of  any  one  particular  test  and  to  avoid  the 
potential  conflict  between  test  program  techni- 
cal objectives  and  milestone  schedule  objec- 
tives. A related  problem  is  that  of  the  almost 
universal  interpretation  of  test  hardware  fail- 
ures as  being  “all  bad."  This  interpretation  is 
due  in  part  to  the  high  visibility  environment  in 


which  testers  work.  The  assessment  and  ulti- 
mate trade  off  of  test  objectives  vs  risk  factors — 
particularly  during  the  early  stages  of  a pro- 
gram— must  necessarily  include  the  potential 
for,  and  the  consequences  of,  various  types  of 
failure.  The  “right-to-fail”  (in  the  test  cell) 
should  be  recognized  as  an  inherent  and  valu- 
able part  of  the  typical  test  and  evaluation  proc- 
ess. Aside  from  the  fact  that  the  basic  intent  of 
some  tests  is  to  demonstrate  failure  modes  and 
limits  so  that  the  designs  can  be  refined,  a test 
event  that  encounters  an  unforeseen,  unique 
type  of  failure  mode  may  provide  meaningful 
information  of  greater  value  to  the  program 
than  does  information  from  a typical  test  de- 
signed to  ascertain  or  prove  some  particular  de- 
sign capability. 

OBSERVATIONS 

In  summary,  there  appear  to  be  a number 
of  guide  posts  in  meeting  the  challenges  in  fu- 
ture aircraft  engine  development. 

• There  do  not  appear  to  be  any  short- 
cuts in  the  development  process,  and 
the  requirements  are  getting  tougher. 

• Advanced  technology  demonstrator 
programs  are  a key  element  in  main- 
taining technological  leadership  and 
are  the  building  blocks  for  new  appli- 
cations. 

• Continued  investment  in  testing  and 
test  measurement  capabilities  is  abso- 
lutely necessary  to  meet  future  techni- 
cal challenges. 

• A better  job  must  be  done  in  establish- 
ing and  interpreting  the  "design-to" 
and  “test-to"  requirements  to  insure 
an  adequate  evaluation  program. 

• Improvement  in  the  ability  to  translate 
in-service  usage  objectives  into  mean- 
ingful and  accurate  test  requirements 
is  needed. 

Finally,  recognition  must  be  given  the  im- 
portance of  “successful  failures.”  There  is  no 
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substitute  for  good  Judgment,  team  work,  and 
management  wisdom — and  a little  bit  of  luck — 
if  we  hope  to  continue  to  improve  the  overall 
test  and  evaluation  process. 
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RELIABILITY  AND  MAINTAINABILITY 
IN  THE  ACQUISITION  PROCESS 
DOD  Directive  5000.x 

by 

Colonel  Ben  H.  Swett,  USAF 


An  effort  to  establish  overall  Department  of  Defense  (DOD)  policy  on  how  reliability  and 
maintainability  should  be  addressed  during  each  phase  of  the  acquisition  process  is  underway. 
The  “for  comment”  draft  of  a new  5000'Series  directive,  DODD  5000.x,  entitled  “Reliability 
and  Maintainability  (R&M)  of  Systems  and  Equipment,”  was  released  to  the  DOD  Compo- 
nents and  defense  industry  associations  in  April  1977.  In  this  article  the  author  discusses  the 
provisions  of  DODD  50()0.x,  the  underlying  rationale,  and  some  of  the  comments  received. 
Formal  coordination  and  implementation  of  this  directive  are  expected  early  in  1978. 


PURPOSE 

The  purpose  of  DODD  5000.x  is  to  im- 
prove the  life  cycle  cost  effectiveness  of  systems 
and  equipment  acquired  by  the  Department  of 
Defense.  Major  goals  are  to  increase  opera- 
tional readiness  and  effectiveness,  reduce  main- 
tenance and  logistic  support  cost,  and  reduce 
acquisition  cost  and  schedule.  Phase-by-phase 
policy  provisions  are  designed  to  implement  the 
policies  of  the  Office  of  Management  and  Bud- 
get OMB  Circular  A- 109,  and  DOD  Directives 
5000.1,  5000.2  and  5000.3.  Provisions  regard- 
ing Military  Standards  for  reliability  and 
maintainability  are  designed  to  implement  the 
findings  of  the  Defense  Science  Board  Task 
Force  on  Specifications  and  Standards.  Five 
specific  objectives  are  outlined  here. 

OBJECTIVE  1 

Establish  the  reliability  and  maintainability 
characteristics  to  be  managed  as  major  perfor- 
mance parameters  of  all  defense  systems  and 
equipment. 


Reliability  and  maintainability  can  be 
characterized,  defined  and  measured  many  dif- 
ferent ways.  Therefore,  it  is  necessary  to  select 
the  general  reliability  and  maintainability  char- 
acteristics that  link  all  other  performance  re- 
quirements to  operational  readiness,  ef- 
fectiveness and  ownership  costs.  The  selection 
will  provide  a basis  for  performance/life  cycle 
cost/effectiveness  trade-offs.  The  general  relia- 
bility and  maintainability  characteristics  se- 
lect^ must  be  managed  as  quantitative  goals 
and  thresholds  during  system  acquisition  to  en- 
sure sufficient  management  attention. 

OBJECTIVE  2 

Ensure  that  realistic  reliability  and  maintaina- 
bility requirements  are  established  for  all  ac- 
quisition programs  and  achieved  in  all  systems 
fielded  by  the  DOD. 

Emphasis  is  on  realistic,  as  opposed  to  ide- 
alistic, requirements.  Included  are  both  quanti- 
tative reliability  and  maintainability  require- 
ments and  the  series  of  tasks  necessary  to 
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achieve  these  requirements.  It  was  necessary  to 
state  that  reliability  and  maintainability  re- 
quirements must  be  established  for  all  acquisi- 
tion programs  (a  practice  not  now  followed  in 
all  cases).  The  objective  is  to  achieve  reliability 
and  maintainability  requirements  in  Held  ser- 
vice, rather  than  in  theoretical  predictions  or 
laboratory  tests. 

OBJECTIVE  3 

Integrate  reliability  and  maintainability  tasks 
into  the  acquisition  process  by  identifying  the 
basis  for  evaluation  of  reliability  and  maintaina- 
bility achievements  at  each  milestone  decision. 

Assigning  specific  reliability  and  main- 
tainability tasks  and  insuring  task  accomplish- 
ments are  Program  Manager  responsibilities. 
Evaluating  accomplishments  of  the  preceding 
phase  and  plans  for  the  next  phase  are  responsi- 
bilities of  the  program  review  and  decision  au- 
thorities. Both  the  Program  Manager  and  the 
decision  authorities  need  a general  outline  of 
what  should  and  should  not  be  expected  at  each 
milestone.  That  “report  card”  is  the  point  at 
which  reliability  and  maintainability  considera- 
tions can  become  an  integral  part  of  program 
management,  review  and  decisionmaking. 
However,  it  will  not  happen  until  the  overriding 
acquisition  cost  and  schedule  problems  of  the 
Program  Manager  and  higher  authorities  are 
taken  into  account  along  with  reliability  and 
maintainability  considerations. 

OBJECTIVE  4 

Minimize  acquisition  cost  and  schedule,  consis- 
tent with  the  above  objectives. 

Several  comments  recommended  changing 
this  objective  to  read,  “Optimize  acquisition 
cost  and  schedule....”  Such  comments  could  not 
be  accepted.  The  intent  is  to  accomplish  the 
other  objectives  through  more  efficient  invest- 
ment in  reliability  and  maintainability  pro- 
grams, not  by  driving  acquisition  cost  higher 
and  pressing  schedule  beyond  present  bounds. 
There  are  some  specific  places  where  more 
needs  to  be  invested  in  reliability  and  maintain- 
ability than  has  been  usual  in  the  past,  but  the 


net  result  must  be  lower  per  unit  cost  and  more 
rapid  delivery.  Reduction  of  total  cost  and 
schedule  has  not  been  a fundamental  objective 
of  reliability  and  maintainability  programs,  but 
it  must  become  so  in  the  present  cost  and  sched- 
ule environment  if  reliability  and  maintainabil- 
ity programs  and  projects  are  to  survive. 

OBJECTIVES 

Promote  continuing  improvement  in  reliability 
and  maintainability  engineering  and  assurance 
technologies  to  meet  current  and  future  system 
program  reliability  and  maintainability  re- 
quirements. 

Some  comments  were  that  the  provisions 
of  DODD  SOOO.x  do  not  support  this  objective. 
The  comments  may  have  been  based  on  a misin- 
terpretation of  what  the  objective  implies.  The 
objective  does  not  mean  continuing  refinement 
of  classical  practice,  but  response  to  a new  and 
broader  range  of  responsibilities.  To  meet  the 
provisions  of  DODD  5000.x,  Program  Manag- 
ers and  higher  authorities  will  need  different 
kinds  of  assistance  from  reliability  and  main- 
tainability staff  than  has  been  received  in  the 
past.  Calling  for  more  than  one  reliability  char- 
acteristic and  more  than  one  maintainability 
characteristic  will  create  challenges  for  alloca- 
tion, prediction,  design,  test  and  evaluation  of 
those  requirements.  This,  in  turn,  will  create 
the  need  for  innovation  in  the  reliability  and 
maintainability  specialties  and  underlying 
technologies. 

APPLICABILITY  AND  SCOPE 

The  5000-series  Directives,  including 
DODD  5000.x,  apply  to  the  Office  of  the  Secre- 
tary of  Defense,  the  Military  Departments  and 
the  Defense  Agencies.  These  directives  apply  to 
contracts  only  insofar  as  they  direct  certain  ac- 
tivities of  the  Program  Manager  or  procuring 
activity.  Nevertheless,  DODD  5000.x  has  been 
developed  with  contractual  implications  in 
mind,  and  with  a conscious  effort  to  incorpor- 
ate accepted  principles  of  sound  business  prac- 
tice. Comments  from  the  defense  industry  asso- 
ciations were  considered  on  an  equal  basis  with 
those  of  the  Military  Departments  and  Defense 
Agencies. 
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Department  of  Defense  Directive  5000.  x 
differs  from  other  5000-series  Directives  in  one 
major  respect;  that  is,  it  applies  to  less-than- 
major  systems,  subsystems  and  equipment  as 
well  as  “Major  defense  systems."  The  expan- 
sion was  necessary  because  reliability  and  main- 
tainability problems  are  not  confined  to  “Major 
defense  systems,”  and  do  not  originate  at  the 
system  level  of  assembly.  As  in  the  story  of  the 
war  that  was  lost  “for  the  want  of  a horse-shoe 
j nail,”  it  is  at  the  lower  levels  of  assembly  that 

! the  correction  of  most  reliability  and  maintain- 

f ability  problems  begins. 

I 

I DEFINITIONS 

The  “for  comment”  draft  of  DODD 
5(XX).x  included  eleven  terms  and  definitions. 
Several  comments  recommended  all  terms  and 
definitions  be  deleted  in  favor  of  the  “accepted 
standard  terminology”  presented  in  MIL-STD- 
i 721.  That  was  not  done,  because  standard  relia- 

I bility  and  maintainability  terminology  does  not 

support  the  performance/cost/schedule  or  life 
cycle  cost  trade-offs  required  by  current  DOD 
acquisition  policy,  and  because  Military  Stan- 
dards are  contractual  documents  that  must  re- 
flect, but  do  not  establish,  DOD  policy. 

The  definitions  in  DODD  5000.x  are  in- 
tended to: 

• direct  the  DOD  Components  in  the  se- 
lection of  existing  reliability  and  main- 
tainability terms  that  link  performance 
requirements  to  readiness,  effective- 
ness and  ownership  cost, 

• distinguish  between  government  and 
contractor  responsibilities, 

• outline  what  is  meant  by  an  “Inte- 
grated” test, 

• bring  Qualification  and  Acceptance 
Tests  more  nearly  in  line  with  estab- 
lished auditing  principles,  and 

• establish  a basis  for  projecting  field  ser- 
vice reliability  and  maintainability  val- 
ues from  demonstrated  test  results. 


None  of  these  problems  has  been  solved  by  cur- 
rent policy  or  practice.  Current  DOD  policy  re- 
quires clarification  of  terminology,  and  is  be- 
yond the  scope  of  Military  Specifications  and 
Standards.  Therefore,  although  wording 
changes  have  been  incorporated  in  response  to 
specific  comments  on  the  definitions  in  DODD 
5000. X,  it  remains  necessary  to  establish  a few 
basic  distinctions  in  reliability  and  maintaina- 
bility terminology  as  a matter  of  DOD  policy. 

Definitions  of  Major  Reliability 

and  Maintainability  \ 

Characteristics  with  Commentary 

Department  of  Defense  Directive  5000.x  j 

does  not  prescribe  a set  of  detailed  terms  and  | 

definitions  for  all  kinds  of  systems  and  equip-  i 

ment.  The  directive  defines  those  general  relia-  ] 

bility  and  maintainability  characteristics  that 
must  be  addressed  as  major  performance  re- 
quirements, and  then  directs  the  DOD  Compo- 
nents to  select  (or  define)  appropriate  terms  for 
each  of  those  characteristics,  as  the  characteris- 
tics apply  to  specific  systems  and  equipment.  In 
essence,  DODD  5000.x  distinguishes  between 
reliability  and  maintainability  as  measures  of 
effectiveness  and  as  factors  of  ownership  cost. 

• Mission  Reliability  will  include  only 
missions  or  mission  time,  and  only 
those  failures  that  are  mission-critical, 
at  a stated  level  of  assembly. 

Mission  reliability  is  a factor  of  mission  ef- 
fectiveness and  must  be  specified  as  a require- 
ment for  all  systems  and  equipment.  Mission 
reliability  has  little  relation  to  availability  or 
operational  readiness,  because  the  system  is  as- 
sumed to  be  ready  at  the  start  of  each  mission. 

Mission  reliability  has  little  relation  to  owner 
ship  cost,  because  mission  reliability  counts 
only  mission-critical  failures  that  occur  during 
the  course  of  a mission.  Redundant  design  and 
alternate  modes  of  operation  increase  mission 
reliability  but  also  increase  system  complexity, 
total  parts  count,  and  acquisition  cost.  Com- 
plex systems  with  a high  parts  count  require 
more  maintenance  and  more  spare  parts  in  field 
service,  because  there  are  more  parts  to  fail.  Ef- 
forts to  increase  mission  reliability  are  directly 
related  to  higher — not  lower — life  cycle  cost. 
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• Maintenance  Reliability  will  include 
all  potentially  useful  operation  and  all 
failure  indications,  for  a stated  level  of 
assembly  and  all  of  its  component 
parts. 

Maintenance  reliability  is  a new  term  that 
refers  to  reliability  as  a factor  of  system  demand 
for  maintenance  and  logistic  support.  Since  ev- 
ery part  can  create  a demand  for  maintenance, 
maintenance  reliability  counts  every  failure  in- 
dication, whether  or  not  it  is  mission-critical, 
whether  or  not  it  is  subsequently  confirmed  as  a 
“material  failure,”  and  regardless  of  redun- 
dancy in  system  design.  Since  failure  indica- 
tions initiate  maintenance  regardless  of  time  of 
occurence,  maintenance  reliability  counts  all 
potentially  useful  operation  and  not  merely 
missions  or  mission  time.  Maintenance  reliabil- 
ity, no  matter  how  defined  and  measured,  must 
answer  the  question:  “How  often  will  this  sys- 
tem require  maintenance”? 

When  “failure  indication”  is  defined  by  de- 
tailed performance  requirements,  maintenance 
reliability  becomes  the  link  between  those  re- 
quirements and  the  effects  of  those  require- 
ments on  availability,  operational  readiness, 
maintenance  manpower  requirements  and  lo- 
gistic support  cost.  Redundant  design  and  alter- 
nate modes  of  operation  do  not  increase,  but 
rather  decrease  maintenance  reliability.  One 
way  to  improve  maintenance  reliability  is  to 
build  simpler  systems  having  a lower  parts 
count.  Thus,  both  acquisition  cost  and  cost  of 
ownership  are  reduced.  Maintenance  reliabil- 
ity, not  mission  reliability,  is  directly  related  to 
lower  system  life  cycle  cost. 

The  introduction  of  maintenance  reliabil- 
ity as  a separate  and  distinct  reliability  charac- 
teristic generated  much  comment  and  con- 
siderable resistance.  Apparently  a number  of 
people  do  not  realize  that  investment  in  reliabil- 
ity programs  cannot  be  justified  on  the  basis  of 
improved  readiness  or  reduced  ownership  cost 
when  only  mission  reliability  is  specified  as  a 
system  performance  requirement. 

“Material  reliability”  was  proposed  as  a 
substitute  for  “maintenance  reliability”  on  the 


basis  that  it  is  a well-recognized  term.  T his  pro- 
posal was  not  acceptable,  because  mat  .-rial  reli- 
ability only  counts  “.^'nfirmed  mate'-ial  failures 
(faults).”  In  some  current  w,,uipment,  85  per- 
cent of  the  Built-In-Test  failure  indications  are 
not  confirmed  as  material  failures.  Hence 
mean-time-between-failure  (Confirmed)  ex- 
ceeds mean-time-between-failure  (Indicated) 
by  a factor  between  6 and  7.  That  is  why 
projections  of  availability,  readiness,  manpower 
and  total  logistics  cost  tend  to  be  optimistic 
when  based  on  demonstrated  material  reliabil- 
ity. However,  material  reliability  values  are 
needed  for  the  planning  of  spare  parts  invento- 
ries. Future  versions  of  DODD  5000.x  will  note 
that  material  reliability  may  be  specified  as  a 
major  reliability  characteristic,  in  addition  to, 
but  not  instead  of,  mission  and  maintenance  re- 
liability. 

• System  Maintainability  will  include 
all  system  (clock)  downtime  from  ini- 
tiation to  completion  of  maintenance, 
excluding  delays  not  dependent  upon 
system  design  characteristics. 

Maintainability,  as  measured  in  clock 
time,  is  a factor  of  system  availability  and  oper- 
ational readiness.  Since  maintainability  is  a sys- 
tem design  characteristic,  it  cannot  include 
maintenance  delays  for  such  things  as  towing, 
getting  personnel  to  and  from  the  job,  waiting 
for  spare  parts  or  technical  data,  etc.  And  since 
maintainability  is  a factor  of  system  availabil- 
ity, it  cannot  include  clock  time  for  off-system 
maintenance  of  detached  components  when  the 
system  itself  is  operationally  ready.  Mainte- 
nance reliability  asks,  “How  often  will  this  sys- 
tem require  maintenance”?  System  main- 
tainability asks,  “How  long  will  the  system  be 
down  when  it  does  require  maintenance”? 

The  “for  comment”  draft  of  DODD 
5000.x  called  out  “mean-time-to-repair 
(MTTR)”  for  this  system  maintainability  char- 
acteristic, when  the  standard  definition  of 
maintainability — which  is  the  probability  that 
maintenance  will  be  completed  within  a speci- 
fied amount  of  time — would  have  been  equally 
acceptable.  The  point  is  that  maintainability  is  a 
factor  of  availability  and  thus  of  operational  ef- 
fectiveness; it  is  not  an  accurate  measure  of 
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cost,  because  manpower  and  material  costs  of 
ofT-system  maintenance  and  repair  are  not 
taken  into  account.  That  is  why  the  use  of 
quick-remove-and-replace  components  im- 
proves maintainability  and  availability,  but 
does  not  significantly  reduce  total  maintenance 
or  logistic  support  costs.  Also  that  is  why 
investment  in  maintainability  programs  cannot 
be  justified  on  the  basis  of  ownership  cost  re- 
duction when  only  the  classic  definition  of 
maintainability  is  specified  as  a system  design 
requirement. 

• Maintenance  Manpower  will  include 
all  maintenance  personnel  or  all  main- 
tenance manhours  required  to  support 
a given  system  at  all  prescribed  levels 
of  maintenance  and  repair. 

Maintenance  manpower  is  one  of  the  larg- 
est factors  of  ownership  cost  for  any  system  that 
requires  maintenance.  This  factor  is  sensitive  to 
two  reliability  and  maintainability  characteris- 
tics— maintenance  reliability  and  maintenance 
manhours  per  maintenance  action — and  it  does 
not  go  away  when  on-system  maintenance  has 
been  minimized  in  the  interests  of  greater  sys- 
tem availability.  Maintenance  manpower  is 
now  being  addressed,  for  some  systems,  by  such 
terms  as  “maintenance  manhours  per  flying 
hour  (MMH/FH)” — which  is  not  an  accurate 
measure  because  it  combines  the  effects  of  vari- 
ation in  flying  hours  per  month,  maintenance 
actions  per  flying  hour  and  manhours  per  main- 
tenance action,  without  distinguishing  among 
them.  The  draft  DODD  5000.  x called  out 
“maintenance  manning  ratio”  for  this  charac- 
teristic— only  to  have  numerous  comments 
pioint  out  that  it  is  better  to  allocate  manhours 
rather  than  fractions  of  people  as  requirements 
for  subsystems  and  equipment.  However  it  is 
measured,  until  this  characteristic  is  addressed 
as  an  integral  part  of  the  Program  Manager’s 
responsibilities,  there  will  be  little  control  over 
maintenance  manpower  cost  in  field  service 
during  the  course  of  system  acquisition. 

• Maintenance  Material  Cost  will  in- 
clude all  replacement  parts  and  com- 
ponents required  to  support  a given 
system  at  ail  prescribed  levels  of  main- 
tenance and  repair 


Maintenance  material  cost  is  sensitive  to 
two  reliability  and  maintainability  characteris- 
tics— material  reliability  and  average  parts 
cost.  Maintenance  material  cost  is  sensitive  to 
certain  design  policies  and  strategies — but  in 
the  wrong  direction.  For  example,  high- 
reliability  parts  typically  cost  from  twice  to  ten 
times  as  much  as  the  commercial-grade  equiva- 
lents, but  these  parts  do  not  always  last  twice  to 
ten  times  as  long.  Since  less  than  ten  percent  of 
the  parts  typically  cause  more  than  ^ percent 
of  the  failures  in  field  service,  mandatory  use  of 
100  percent  high-reliability  parts  becomes  ques- 
tionable. In  the  same  way,  the  use  of  sealed 
components  as  a way  to  avoid  maintenance- 
induced  failures  means  that  the  entire  compo- 
nent must  be  removed,  shipped,  repaired,  re- 
turned and  replaced  every  time  the  weakest 
part  in  the  component  experiences  a failure. 
The  aggregate  impact  of  such  policies  and  de- 
sign strategies  does  not  show  up  unless  some- 
thing like  average  material  cost  per  repair  is 
managed  as  a system  design  characteristic.  The 
Draft  DODD  5000.x  did  not  mention  the  main- 
tenance material  cost  characteristic,  but  several 
comments  pointed  out  the  need  for  it  or  a simi- 
lar characteristic. 

• Availability  is  the  probability  that  a 
system  will  be  in  specified  operational 
condition  if  called  upon  to  perform  its 
mission  at  any  random  point  in  time. 

Availability  was  not  selected  as  a major  re- 
liability and  maintainability  characteristic, 
even  though  it  is  in  wide  usage  and  a number  of 
comments  recommended  that  it  be  included. 
Availability  can  be  derived  from  maintenance 
reliability  and  maintainability  (downtime). 
Further,  availability  is  not  directly  measurable 
during  system  acquisition.  Future  versions  of 
DODD  5000.x  will  note  that  the  term  availabil- 
ity can  be  used  instead  of  system  maintain- 
ability, but  not  instead  of  maintenance  reliabil- 
ity, because  the  latter  is  necessary  as  a factor  of 
ownership  cost. 

RELEVANT  VS  CHARGEABLE 

Use  of  the  term  “nonrelevant  failure”  has 
caused  a great  deal  of  confusion.  In  some  cases. 
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the  Program  Manager  has  deleted  a significant 
percentage  of  the  failures  actually  observed 
during  a reliability  test  because  the  failures 
were  caused  by  factors  beyond  the  control  of 
the  contractor.  Thereafter  other  agencies  found 
that  operational  readiness,  effectiveness  and 
ownership  cost  calculations  based  on  demon- 
strated test  results  were  optimistic.  In  other 
cases,  the  Program  Manager  has  deleted  only 
those  failures  that  were  obviously  test-peculiar. 
In  these  cases  the  contractor  was  penalized  for 
failures  over  which  he  did  not  have  control. 
None  of  these  choices  is  an  acceptable  solution 
to  the  problem  of  analyzing  and  using  reliability 
test  data. 

The  DODD  5000.x  attempts  to  resolve  the 
dilemma  by  recognizing  that  reliability  test  re- 
sults must  be  used  for  two  different  purposes; 
projection  of  field  service  reliability  values,  and 
determination  of  contract  compliance.  The  di- 
rective establishes  a clear  distinction  between 
relevant  failures  and  chargeable  failures  and  ex- 
pands on  that  distinction  to  include  clearly  de- 
fined responsibilities  for  maintenance  down- 
time, manpower  and  materiel  costs. 

• Relevant  Failures.  Relevant  failures 
include  all  failures,  downtime  hours, 
maintenance  manhours  and  materiel 
costs  incurred  during  test  that  can  be 
expected  to  occur  in  field  service. 

• Nonrelevant  Failures.  Sonrelevant 
failures  include  only  those  failures, 
downtime  hours,  maintenance  man- 
hours and  materiel  costs  incurred  dur- 
ing test  that  are  caused  by  a condition, 
external  to  the  equipment  under  test, 
that  is  not  encountered  in  field  ser- 
vice. 

“Relevant”  and  “Nonrelevant”  refer  to 
field  service  and  to  the  responsibility  of  the  gov- 
ernment Program  Manager  or  procuring  activ- 
ity. The  terms  therefore  are  equally  applicable 
to  Government  Furnished  Equipment  and 
Contractor  Furnished  Equipment  components 
of  the  item  under  test.  If  a significant  portion  of 
field  service  reliability  and  maintainability 


problems  are  found  to  originate  in  Government 
Furnished  Equipment  components  of  the  sys- 
tem, it  is  the  responsibility  of  the  Program 
Manager  to  have  the  components  repaired  or 
replaced. 

Note  that  all  test  data  are  considered  rele- 
vant unless  and  until  such  data  are  proved  to  be 
test-peculiar.  Only  then  may  this  test  data  be 
deleted  from  projections  of  field  service  reliabil- 
ity and  maintainability  values. 

• Chargeable  Failures.  Chargeable  fail- 
ures are  those  relevant  failures,  down- 
time hours,  maintenance  manhours 
and  materiel  costs  incurred  during  test 
which  are  caused  hy  any  of  the  goods 
or  services  provided  by  a given  con- 
tractor. 

• Sonchargeable  Failures.  Noncharge- 
able  failures  incurred  during  test  in- 
clude only  those  relevant  failures, 
downtime  hours,  maintenance  man- 
hours and  materiel  costs  which  are 
caused  hy,  and  are  dependent  upon,  a 
condition  previously  stipulated  as  not 
within  the  responsibility  of  a given 
contractor. 


“Chargeable”  and  “Nonchargeable”  are 
subsets  of  “relevant.”  If  test  data  are  not  rele- 
vant to  field  service,  the  failures  are  not  charge- 
able to  the  contractor.  The  limits  of  contractor 
responsibility  must  be  defined  in  advance  and 
stipulated  as  scoring  rules  for  any  test  used  to 
determine  compliance.  “Chargeable”  does  in- 
clude the  effects  of  contractor-furnished  soft- 
ware, support  equipment,  training,  operation, 
maintenance  or  repair  procedures,  etc.,  as  well 
as  the  hardware  components  of  the  product. 
Where  goods  and  services  supplied  by  more 
than  one  contractor  are  involved  in  the  same 
test,  “chargeable"  may  be  mor;  specifically  de- 
fined to  distinguish  between  the  responsibilities 
of  the  various  contractors.  The  term  “non- 
chargeable” cannot  be  used  to  delete  test  data 
from  projections  of  field  service  reliability  and 
maintainability  values. 
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INTEGRATED  TEST 

One  member  of  a recently  established  Pro- 
gram Office  surveyed  all  applicable  Military 
Regulations,  Specifications  and  Standards  to 
see  how  many  tests  were  required  for  an  avion- 
ics subsystem.  Thirty-nine  separate  and  distinct 
kinds  of  test  were  found,  some  of  which  had  to 
be  run  several  times  during  the  course  of  the  ac- 
quisition program.  This  number  did  not  include 
tests  required  for  higher  or  lower  levels  of  as- 
sembly. From  similar  experience  throughout 
DOD,  it  appears  that  a larger  number  of  tests  or 
test  programs  are  not  needed.  What  is  needed  is 
a rational  and  cost-effective  approach  to  testing 
in  general. 

Such  an  approach  for  major  weapon  sys- 
tems, at  the  system  level  of  assembly,  is  speci- 
fied in  DODD  5000.1,  5000.2  and  5000.3. 
However,  the  Military  Standard  tests  for  sub- 
systems and  equipment  are  fragmented.  Perfor- 
mance is  tested  under  ambient  conditions,  with- 
out consideration  of  reliability  or  maintainabil- 
ity. Environmental  qualification  tests  apply 
high — and  in  some  cases,  extreme — stress  lev- 
els, without  equipment  operation  or  without 
recording  equipment  reliability  under  stress. 
Reliability  tests  demand  extensive  equipment 
operating  time,  under  conditions  that  do  not 
adequately  simulate  the  operational  environ- 
ment, and  under  a definition  of  “failure”  that 
does  not  begin  to  count  all  the  failures  defined 
by  specified  performance  tolerance.  Maintaina- 
bility tests  are  not  required  for  subsystems  or 
equipment,  and  none  of  the  other  tests  require  a 
record  of  maintainability  data.  Result;  ^uip- 
ment  can  go  through  all  the  standard  tests  with- 
out revealing  many — let  alone  most — of  the 
problems  the  equipment  will  display  in  field 
service. 

Reliability  tests  are  notorious  for  taking  a 
long  period  of  time  or  a large  number  of  sam- 
ples. Department  of  Defense  acquisition  policy 
calls  for  test  results  as  inputs  to  the  production 
decision.  But  the  Program  Manager  does  not 
have  many  hours  of  test  time  between  the  end  of 
Full-Scale  Engineering  Development  and  the 
start  of  production.  Further,  the  Program  Man- 
ager cannot  afford  a large  number  of  samples 


for  reliability  testing.  Present  cost  and  schedule 
constraints  dictate  a reduction  in  the  amount  of 
time  required  for  reliability  testing.  Many  de- 
tailed provisions  scattered  throughout  DODD 
5000.x  are  designed  to  do  just  that. 

Although  specific  tests  must  be  tailored  to 
the  mission  and  operational  environment  of  the 
equipment  under  test,  a few  principles  of  test 
cost-effectiveness  apply  in  all  cases.  The 
DODD  5000.x  specifies  one  integrated  test 
wherein  appropriate  environmental  stresses  are 
combined  insofar  as  practical,  and  where  speci- 
fied performance,  reliability  and  maintainabil- 
ity characteristics  are  tested  simultaneously. 

PROVISIONS  OF  INTEGRATED  TEST 

• Test  conditions  shall  reproduce  mea- 
sured or  predicted  environmental 
stresses  in  field  service,  based  on  a 
trade  off  between  maximum  test  real- 
ism and  affordable  test  facilities. 

• Test  procedures  shall  simulate  opera- 
tional use  and  performance  monitoring 
of  the  equipment  under  test. 

• Reliability  and  maintenance  data  shall 
be  recorded  during  the  test,  and  testing 
shall  be  of  sufficient  duration  to  mea- 
sure demonstrated  reliability  and 
maintainability  values. 

• Separate  tests  that  duplicate  portions 
of  the  more  realistic  integrated  test 
shall  not  be  required. 

Integrated  testing  serves  two  purposes: 
Correlation  between  test  results  and  field  ser- 
vice data  is  increased  by  increasing  test  realism, 
and  the  time  required  for  reliability  tests  is  re- 
duced by  increasing  the  rate  at  which  any  given 
number  of  failures  will  be  discovered.  TTie  test 
time  reduction  is  roughly  proportional  to  the 
degree  that  present  reliability  test  conditions 
are  less  failure-producing  than  integrated  test 
conditions.  The  DODD  5000.x  does  not  state 
how  to  structure  an  integrated  test  for  any  sys- 
tem, subsystem  or  equipment.  The  directive 
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states  principles  of  test  cost-effectiveness  as  pol- 
icy for  the  direction  in  which  the  DOD  compo- 
nents must  move  if  reliability  and  maintainabil- 
ity test  programs — and  more  specifically,  labo- 
ratory test  programs — are  to  become  truly  cost- 
effective. 

INDEPENDENT  TEST  AND 
EVALUATION 

Even  though  the  requirement  for  indepen- 
dent test  and  evaluation  was  specifically  limited 
to  preproduction  qualification  tests  and 
production  acceptance  tests,  this  provision  of 
DODD  5000.x  generated  a storm  of  protest. 
The  DOD  components  said  compliance  would 
be  too  expensive.  The  defense  industries  do  not 
like  others  testing  their  equipment.  The  com- 
ments are  correct  in  one  respect:  the  issue  of 
independent  test  and  evaluation  overshadows 
DODD  5000.x — and  may  overshadow  DODD 
5000.1,  5000.2  and  5000.3  combined.  The  issue 
was  surfaced  in  response  to  the  management 
principles  incorporated  in  DODD  5000.3, 
“Test  and  Evaluation,”  (including  the  standard 
auditing  principle  that  says  no  one  should  be 
asked  to  evaluate  his  own  product),  and  a ques- 
tion from  the  House  Appropriations  Commit- 
tee that  asked,  “Is  it  true  that  defense  contrac- 
tors test  their  own  products  on  behalf  of  the 
government”? 

It  is  true.  That  is  the  almost-universal 
practice  in  current  reliability  and  maintainabil- 
ity programs.  Government  inspectors  are  re- 
sponsible for  monitoring  qualification  and  ac- 
ceptance tests  at  the  larger  plants.  Smaller 
plants  and  vendor  facilities  are  visited  only  oc- 
casionally. Nevertheless,  once  a product  passes 
these  tests  and  is  placed  on  a Qualified  Parts 
List  all  higher  tier  contractors  have  to  use  the 
product  unless  a waiver  can  be  justified  (and 
obtained).  If  the  higher-tier  contractors  receive 
faulty  parts,  neither  these  contractors  nor  the 
Program  Managers  have  much  recourse  when 
the  parts  already  have  been  accepted  as  compli- 
ant with  government  specifications. 

Department  of  Defense  Directive  5000.x 
could  not  resolve  the  inherent  conflict  of  inter- 
est involved  in  having  every  producer  test  his 


own  products  on  behalf  of  the  government.  The 
directive  merely  defined  independent  test  and 
evaluation  as  “a  test  conducted  by  a gov- 
ernment or  commercial  agency  having  no 
vested  interest  in  the  outcome  of  the  test,”  and 
noted  that  independent  test  and  evaluation  is 
called  for  by  DODD  5000.3.  Then  DODD 
5000.x  provided  three  options — in  priority — 
for  the  testing  of  subsystems  and  equipment. 

1.  Independent  T&E  is  always  preferred 
for  Preproduction  Qualification  and 
Production  Acceptance  Tests,  but  it  is 
not  always  feasible  nor  cost-effective. 

2.  The  second  choice  is  to  have  prime 
contractors  test  the  products  of  their 
subcontractors,  and  subcontractors 
test  the  products  of  their  vendors,  un- 
der government  surveillance. 

3.  The  third  choice  is  to  have  the  pro- 
ducer conduct  Preproduction  Qualifi- 
cation or  Production  Acceptance  Tests 
of  his  own  product  under  strict  govern- 
ment surveillance  and  independent 
evaluation  of  the  test  results. 

The  options  were  considered  to  be  the 
most  logical  compromise  between  further  sup- 
port of  the  current  practice — which  essentially 
reverses  these  priorities — and  the  unknown 
cost  impact  of  requiring  truly  independent  test 
and  evaluation  across  the  board.  The  directive 
leaves  selection  from  among  these  priorities  to 
the  Program  Manager  and  the  degree  of  en- 
forcement to  higher  program  review  and  deci- 
sion authorities.  The  fundamental  question  of 
independent  test  and  evaluation  vs  inherent 
conflict  of  interest  is  a matter  for  the  executive 
levels  of  the  DOD  and  the  United  States  Con- 
gress. 

TEST  REALISM 

Classical  definitions  of  reliability  and 
maintainability  all  contain  the  caveat:  “under 
specified  conditions.”  The  definitions  of  major 
reliability  and  maintainability  characteristics  in 
DODD  5000.x  give  recognition  to  the  fact  that 
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reliability  and  maintainability  data  are  applica- 
ble only  to  a specified  set  of  conditions.  There 
are  two  sets  of  conditions — laboratory  test  and 
field  service.  The  relation  between  these  two 
conditions  must  be  known  or  estimated  if  quan- 
titative reliability  and  maintainability  re- 
quirements and  achievements  are  to  be  useful  as 
decision  information.  Why  measure  reliability 
and  maintainability  values  in  a laboratory,  or 
use  laboratory  tests  to  determine  contractual 
compliance,  if  one  has  no  idea  of  the  relation 
between  laboratory  tests  and  field  service  con- 
ditions? 

Test  Realism:  The  degree  to  which  a 
specified  set  of  test  conditions  and  pro- 
cedures simulates  a specified  opera- 
tional environment. 

This  is  the  reciprocal  of  the  “Environmen- 
tal K Factor”  long  used  to  adjust  parts  failure 
ratings  for  different  applications.  As  shown 
here,  the  degree  of  test  realism  would  be  less 
than  1.0  if  the  test  conditions  were  less  failure- 
producing  than  the  operational  environment; 
and,  greater  than  1.0  for  accelerated  or  over- 
stress testing.  Realism  would  have  to  be  ob- 
tained by  comparing  demonstrated  values  for 
the  same  reliability  and  maintainability  charac- 
teristics, using  the  same  definitions  and  the 
same  or  similar  kinds  of  equipment,  under  the 
two  sets  of  conditions  and  procedures:  Labora- 
tory test  and  field  service. 

An  estimated  degree  of  test  realism  is  nec- 
essary for  the  translation  of  field  se  ice  relia- 
bility and  maintainability  requirements  into 
specified  values  for  a test  that  does  not  perfectly 
simulate  field  service.  Also  an  estimate  of  test 
realism  is  necessary  as  an  adjustment  to  the  val- 
ues demonstrated  during  test  when  projecting 
the  values  expected  in  field  service.  Until  labo- 
ratory test  results  can  be  related  to  field  service 
reliability  and  maintainability  values,  the  re- 
sults are  useless  as  inputs  to  operational  readi- 
ness, effectiveness  or  ownership  cost  models. 
Even  though  establishing  rough  estimates  of 
test  realism  will  require  effort  on  the  part  of  the 
DOD  Components,  until  it  is  done,  reliability 
and  maintainability  or  integrated  tests  cannot 


be  justified  on  the  basis  of  cost-effectiveness  be- 
cause there  is  no  way  to  ascertain  the  cost- 
effectiveness. 

POLICY 

The  draft  DODD  5000.x  has  been  criti- 
cized as  having  gone  into  too  much  detail  for  a 
DOD  Directive.  The  criticism  may  have  merit. 
However,  there  is  a pattern  to  the  comments 
that  makes  it  impossible  just  to  shorten  the  Di- 
rective. First,  every  provision  in  the  draft  was 
designed  to  correct  a specific  problem  that  has  a 
large  and  widespread  impact  on  readiness,  ef- 
fectiveness, ownership  cost,  and  acquisition 
cost  or  acquisition  schedule.  Second,  those  who 
have  encountered  one  of  these  problems  almost 
invariably  approve  of  the  provision  designed  to 
correct  it.  Third,  those  who  have  not  encoun- 
tered a particular  problem — or  have  not  recog- 
nized it — almost  invariably  consider  the  provi- 
sions to  be  unnecessary  detail.  Except  for  the 
provision  concerning  independent  test  and  eval- 
uation, where  the  uproar  was  well-nigh  unani- 
mous, the  comments  to  other  provisions  tend  to 
cancel  each  other.  Future  versions  of  DODD 
SOOO.x  will  be  shorter  and  more  pointed — but 
will  be  accompanied  by  DOD  Instruction 
5000.x  that  will  include  more  explanation  than 
is  appropriate  in  a Directive. 

GENERAL 

It  was  necessary  to  state  the  policy  that  re- 
liability and  maintainability  characteristics  will 
be  considered  major  performance  parameters  of 
all  defense  systems,  subsystems  and  items  of 
equipment.  Presently  these  characteristics  are 
not  so  considered.  It  was  necessarv  to  state  the 
minimum  set  of  reliability  and  maintainability 
characteristics  because  those  characteristics 
now  selected  as  performance  parameters  often 
fail  to  support  required  trade  offs  or  life  cycle 
cost  reduction.  It  was  necessary  to  require  that 
each  reliability  and  maintainability  characteris- 
tic be  address^  at  each  program  milestone  de- 
cision and  contractual  review.  Too  often  these 
characteristics  are  ignored.  Othe'  general  pol- 
icy provisions  and  the  applicable  underlying  ra- 
tionale are  as  follows: 
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Speciffc  Reliability  and  Maintainabiiity 
tasks  are  to  be  defined  and  documented  as  an 
integral  part  of  both  government  and  contrac- 
tual program  plans,  rather  than  as  an  addi- 
tional, sideline  “R&M  Program."  This  action  is 
to  ensure  that  these  tasks  get  the  necessary  de- 
gree of  management  attention  throughout  the 
acquisition  process. 

Quantitative  Reiiabiiity  and  Maintaina- 
biiity Requirements  and  Achievements  in  DOD 
program  documents — such  as  the  initial  state- 
ment of  an  operational  requirement,  the  Deci- 
sion Coordinating  Paper,  Program  Memoran- 
dum, and  equivalent  management  documents 
within  the  DOD  components — are  to  use  the 
terms  and  definitions  by  which  reliability  and 
maintainability  achievements  will  be  measured 
in  field  service.  There  has  been  unnecessary 
confusion  between  field  service  and  contractual 
definitions  of  the  same  reliability  and  maintain- 
ability terms.  The  confusion  results  in  numbers 
of  wide  variation.  Explanation  should  not  have 
to  be  made  to  the  Congress  about  why  the  num- 
bers measured  in  early  field  service  do  not  mean 
the  same  thing  as  the  numbers  stated  in  the 
Decision  Coordinating  Paper. 

Quantitative  Reiiabiiity  and  Maintaina- 
biiity Values  in  Contracts  differ  from  field  ser- 
vice values.  It  is  not  feasible  to  hold  the  contrac- 
tor responsible  for  failures,  downtime  hours, 
maintenance  manhours  and  materiel  costs  over 
which  he  has  no  control.  Laboratory  test  re- 
quirements differ  from  field  service  require- 
ments because  a set  of  test  conditions  does  not 
exist  that  can  accurately  simulate  the  field  ser- 
vice environment.  Design  requirements  differ 
from  test  requirements  by  the  margins  used  to 
establish  a specified  degree  of  statistical  confi- 
dence or  decision  risk.  It  is  not  a matter  of  using 
one  and  only  one  number  for  each  reliability 
and  maintainability  characteristic.  The  numeri- 
cal traceability  of  reliability  and  maintainability 
requirements  and  achievements  must  be  estab- 
lished through  a form  of  audit  trail.  Depart- 
ment of  Defense  Directive  5000.x  does  not  tell 
the  DOD  components  how  to  establish  numeri- 
cal traceability,  but  it  does  say  it  must  be  done. 
Also  the  Directive  does  not  indicate  the  major 
factors  that  must  be  included  in  the  audit  trail. 


As  stated  in  several  comments,  some  of  the  cor- 
relation factors  do  not  exist.  It  will  take  time 
and  effort  to  develop  these  factors.  Neverthe- 
less, until  reliability  and  maintainability  re- 
quirements and  achievements  are  numerically 
traceable,  program  review  and  decision  author- 
ities will  not  know  what  any  reliability  or  main- 
tainabiiity number  really  means,  and  neither 
will  the  Program  Manager. 

Theoretical  Reliability  and  Maintainabil- 
ity Predictions  based  on  paper  studies  of  equip- 
ment designs  were  prohibited  as  a basis  for 
source  selection.  The  prohibition  made  a num- 
ber of  people  unhappy,  even  though  it  complied 
with  the  spirit  and  the  letter  of  all  5000-series 
directives-^irectives  that  make  it  clear  that 
major  decisions  are  to  be  based  on  demon- 
strated test  results  rather  than  paper  studies. 
The  prohibition  was  directed  against  two  com- 
mon problems:  First  problem:  the  cases  where 
one  contractor’s  “predicted  mean-time- 
between-failure”  is  competed  against  another’s 
in  the  process  of  source  selection.  The  practice 
establishes  a market — or  an  auction — in  which 
the  contract  goes  to  the  best  predictor.  The  sec- 
ond problem  is  the  practice  of  evaluating  pre- 
dicted reliability  and  maintainability  values 
against  field  service  or  test  requirements,  in- 
stead of  against  design  requirements  derived  via 
an  audit  trail  such  as  is  discussed  here.  The  pro- 
cedure allows  the  design  to  be  accepted  without 
compensation  for  a series  of  adjustment  fac- 
tors— which  means  trouble  later  when  the  de- 
sign is  put  to  test.  In  those  cases  where  it  is 
absolutely  necessary  to  base  source  selection* 
on  paper  predictions  rather  than  on  demon- 
strated test  results  the  procedure  is  allowed  (if 
predictions  are  evaluates  against  design  re- 
quirements and  are  not  used  as  a basis  for  direct 
competition  among  the  bidding  contractors). 

Operational  Readiness,  Effectiveness  and 
Ownersbip/Lite  Cycle  Cost  Models  have  been 
entered  with  contractor’s  predictions.  The  re- 
sults of  these  models  have  been  used  as  a basis 
for  source  selection.  In  addition  to  creating  a 
paper  auction,  this  procedure  leads  to  highly 


• The  major  program  decision  for  the  contractor. 
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optimistic  field  service  estimates.  Thus,  it  was 
necessary  to  state  only  that  those  Held  service 
reliability  and  maintainability  values  estab- 
lished as  requirements  to  be  demonstrated  prior 
to  the  production  decision  are  to  be  used  to  en- 
ter such  models  at  any  point  in  the  program  be- 
fore field  service  reliability  and  maintainability 
values,  based  on  demonstrated  test  results,  are 
available.  Thereafter,  the  models  must  be  up- 
dated by  demonstrated  test  results,  and  the  out- 
puts of  the  models  must  be  available  as  inputs  to 
the  production  decision. 

Integrated  Performance,  Reliability  and 
Maintainability  and  Environmental  Testing 
was  stated  as  a policy  for  all  program  phases. 
This  policy  is  required  only  for  Preproduction 
Qualification  and  Production  Acceptance 
Tests.  The  policy  is  to  ensure  Program  Man- 
ager authorization  to  cut  down  on  duplicative 
or  overlapping  tests.  The  policy  was  not  in- 
tended to  mandate  full-blown  “Mission  profile, 
combined  stress,  integrated  laboratory  tests” 
where  such  tests  are  neither  appropriate  nor 
cost-effective.  Future  versions  of  DODD 
5000.x  will  clarify  this  stipulation. 

Statistical  Estimation  Within  Specified 
Confidence  Limits  was  called  for,  in  addition  to 
hypothesis  test  plans  in  documents  such  as 
MIL-STD-781.  Reason:  hypothesis  tests  can- 
not be  used  to  project  the  reliability  and  main- 
tainability values  expected  in  field  service.  Hy- 
pothesis tests  yield  one  of  two  answers,  “Ac- 
cept” or  “Reject.”  The  true  values  demon- 
strated by  the  test  are  ignored.  Statistical  esti- 
mation techniques  support  the  idea  that  statis- 
tics should  be  the  servant,  rather  than  the 
master,  of  acquisition  cost  and  schedule.  Statis- 
tical estimation  techniques  provide  the  Pro- 
gram Manager  some  confidence  in  the  true  val- 
ues demonstrated  as  the  basis  for  making  a non- 
predetermined  management  decision.  The  Pro- 
gram Manager  then  does  not  have  to  spend  the 
time  necessary  to  fulfill  predetermined 
accept/reject  criteria.  Department  of  Defense 
Directive  SOOO.x  did  not  say  that  is  the  way  it 
should  be  done,  but  there  are  cases  where  the 
Program  Manager  has  to  make  such  a decision. 
In  these  cases  the  Program  Manager  needs  all 
the  help  he  can  get.  The  Program  Manager  does 
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not  need  the  restrictions  imposed  by  enforce- 
ment of  hypothesis  test  plan  assumptions. 

Fixed-Length  Test  Plans  were  given  pre- 
ference ove.  the  “standard  sequential  test 
plans,”  especially  for  preproduction  qualifica- 
tion tests,  for  a number  of  reasons.  First,  all  se- 
quential test  plans  contain  built-in  cost  and 
schedule  uncertainties  that  the  Program  Man- 
ager cannot  control.  Second,  if  program  cost 
and  schedule  is  planned  on  the  basis  of  the  “ex- 
pected decision  point”  in  standard  sequential 
test  plans,  the  inherent  test  time  uncertainty  be- 
comes a built-in  cost  and  schedule  overrun  that 
the  Program  Manager  cannot  control.  (For  ex- 
ample, in  MIL-STD-781  Test  Plan  III,  there  is 
a 10  percent  probability  of  a 100  percent  over- 
run even  though  true  mean-time-between- 
failure  is  equal  to  the  specified  mean-time- 
between-failure.  Third,  Fixed-length  test  plans 
can  be  used  in  conjuction  with  statistical  esti- 
mation techniques.  Sequential  test  plans  can- 
not. Fourth,  while  some  cautionary  notes  with 
regard  to  the  inherent  problems  are  included  in 
the  recent  revision  of  MIL-STD-781,  it  is  not 
the  only  document  that  contains  sequential  test 
plans.  Such  test  plans  are  often  applied  without 
reference  to  cautionary  notes.  It  was  absolutely 
necessary  to  state  that  fixed  length  test  plans 
are  to  be  preferred  for  Preproduction  Qualifica- 
tion Tests  where  the  Program  Manager  cannot 
afford  schedule  uncertainty — much  less  a built- 
in  overrun — and  that  when  sequential  test  plans 
are  used,  program  cost  and  schedule  are  to  be 
planned  on  the  basis  of  “maximum  allowable 
test  time,”  rather  than  “expected  decision 
point.”  Thus  inherent  test  time  uncertainty  re- 
sults in  an  underrun  rather  than  an  overrun. 

Preliminary  Studies.  The  reliability  and 
maintainability  characteristics  of  current  sys- 
tems and  forces  must  be  considered  as  factors  of 
their  operational  readiness,  effectiveness  and 
ownership  cost  when  Mission  Area  Analyses 
are  performed  and  when  the  Mission  Element 
Need  Statements  (MENS)  are  prepared. 
Circular  A- 109  and  E>ODD  5000.1  prohibit  the 
establishment  of  specific  requirements  that 
presuppose  a system  solution  to  the  mission 
need  at  the  onset  of  the  program.  The  MENS 
can  say  about  reliability  and  maintainability 
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only  something  like,  “We  need  a system  with 
twice  the  readiness  and  half  the  maintenance 
personnel  of  the  XXX  and  YYY  systems  now 
being  used  in  this  mission  area.” 

PROGRAM  PHASES 

This  section  of  the  Directive  is  essentially  a 
checklist  for  program  reviews  prior  to  each 
milestone  decision.  Stated  are  the  reliability  and 
maintainability  activities  that  should  be  evalu- 
ated as  accomplishments  of  the  preceeding 
phase.  The  activities  at  each  of  these  reviews 
should,  after  evaluation,  be  addressed  as 
requirements  for  the  following  phase.  This  por- 
tion of  the  directive  is  structured  to  implement 
the  policies  of  OMB  Circular  A- 109  and  the 
other  5000-series  DOD  Directives. 

CONCEPTUAL  PHASE 

In  the  Conceptual  Phase  appropriate  relia- 
bility and  maintainability  characteristics  must 
be  selected  or  defined  for  each  type  of  system 
being  considered  as  a candidate.  Firm  quantita- 
tive requirements  are  prohibited.  The  primary 
concern  must  be  to  ensure  that  the  right  relia- 
bility and  maintainability  characteristics  are  se- 
lected and  defined  in  measurable  terms.  The 
second  concern  must  be  to  ensure  that  a realis- 
tic baseline  is  established  for  the  quantitative  re- 
quirements that  are  to  be  developed  later  in  the 
program.  The  third  concern  must  be  to  prepare 
a baseline  for  estimation  of  technical  risk.  Many 
of  the  problems  that  plague  current  acquisition 
programs  can  be  traced  to  an  idealistic,  over- 
optimistic  or  ill-defmed  baseline  for  the  reliabil- 
ity and  maintainability  requirements. 

Two  values  are  required  for  each  major 
reliability  and  maintainability  charac- 
teristic, The  Hrst  value  shall  be  mea- 
sured from  a similar  system  or  equip- 
ment in  field  service.  The  second  value 
shall  be  a tentative  field  service  re- 
quirement for  the  new  system  or  equip- 
ment. Both  values  shall  use  the  terms 
and  definitions  selected  for  the  new 
system  or  equipment. 

This  provision  is  specifically  intended  to 
avoid  the  practice  of  generating  reliability  and 


maintainability  requirements  by  theoretical 
analysis  of  hypothetical  missions  and  with  little 
or  no  consideration  of  past  and  present  field  ex- 
perience. Also  the  provision  is  intended  to  en- 
sure that  development  planners  obtain  and  use 
field  reliability  and  maintainability  data.  Too, 
the  provision  is  to  ensure  that  tentative  reliabil- 
ity and  maintainability  requirements  are  field 
service  values,  rather  than  attempts  to  compen- 
sate for: 

• degree  of  laboratory  test  realism, 

• the  difference  between  field  service  and 
contractual  definitions,  or 

• any  margins  for  statistical  confidence 
or  decision  risk. 

All  of  these  latter  adjustments  are  to  be  made 
later  in  the  program,  by  the  Program  Office  or 
procuring  activity. 

Tentative  reliability  and  maintainabil- 
ity requirements  shall  be  based  on  the 
sensitivity  of  operational  readiness,  ef- 
fectiveness and  ownership  cost  of  simi- 
lar systems  and  equipment  to  changes 
in  the  numerical  value  of  each  major 
reliability  and  maintainability  charac- 
teristic. 

Tentative  reliability  and  maintainability 
requirements  must  not  be  arbitrary,  and  must 
not  be  set  several  times  higher  than  measured 
field  service  reliability  and  maintainability  val- 
ues from  similar  systems  without  making  clear 
why  such  improvement  is  needed.  For  example, 
the  difference  between  90  and  99  percent  proba- 
bility of  mission  success  increases  required 
mean-time-between-failure  (Critical)  by  an  or- 
der of  magnitude.  That  factor  drives  the  entire 
acquisition  program.  Although  a 9 percent  in- 
crease in  mission  success  may  be  essential,  the 
cost  and  schedule  impact  of  attaining  the  in- 
crease must  be  understood  and  justified  from 
the  beginning.  Since  the  new  system  is  still  in 
the  earliest  stages  of  concept  formulation  at  this 
point,  the  potential  payoff  for  any  tentative  reli- 
ability and  maintainability  requirement  can  be 


estimated  only  by  analysis  of  Held  experience 
with  similar  systems. 

Tentative  reiiability  and  maintainabil- 
ity requirements  shall  be  annotated  as 
being  within  current  state-of-tbe-art  or 
as  requiring  a significant  advance  in 
reliability  and  maintainability  tech- 
nology. 

The  difference  between  measured  field  ser- 
vice reliability  and  maintainability  values  for  a 
similar  system  and  tentative  reliability  and 
maintainability  requirements  for  a new  system 
must  be  used  to  provide  initial  visibility  on  the 
degree  of  technological  advance  being  sought 
for  the  new  system.  We  must  answer  the  ques- 
tion, “Is  this  requirement  technically  feasible”? 
The  answer  must  be  an  estimate  of  technologi- 
cal risk,  based  on  analysis  of  experience  with 
similar  acquisition  programs  and  a review  of 
the  technology  base.  One  or  more  reliability 
and  maintainability  characteristics  may  be  im- 
proved by  a factor  of  2 or  4 or  10  over  the  previ- 
ous generation  of  systems  and  equipment — but 
we  need  to  be  able  to  explain  why  this  is  be- 
lieved to  be  possible,  rather  than  saying,  “Do 
it,”  thus  handing  the  Program  Manager  an  im- 
possible dream. 

DEMONSTRATION  AND 
VALIDATION  PHASE 

The  Demonstration  and  Validation  Phase 
is  where  the  Program  Office  has  to  translate 
tentative  field  service  reliability  and  maintaina- 
bility requirements  into  tentative  contractual 
requirements  and  include  these  requirements  in 
design  solicitations.  Requests  for  alternative  de- 
sign proposals  must  explain  the  reliability  and 
maintainability  characteristics  sought,  and 
must  provide  industry  with  sufficient  informa- 
tion so  that  industry  may  furnish  a meaningful 
response.  Conversely,  each  offeror  must  be  free 
to  propose  his  own  technical  approach,  main 
design  features,  and  alternatives  to  schedule, 
cost  and  capability  goals.  Detailed  government 
specifications  and  standards  are  to  be  avoided 
during  all  program  phases  short  of  Full-Scale 
Engineering  Development  (OMB  Circular  A- 
109).  Therefore,  there  is  a set  of  reliability  and 


maintainability  activities  which  are — and  a 
larger  set  which  are  not — appropriate  during 
the  Demonstration  and  Validation  Phase. 

Design  Solicitations  to  Industry  shall  in- 
clude: 

1 . the  criteria  to  be  used  in  measuring  re- 
liability and  maintainability  achieve- 
ments after  the  system  is  deployed, 

2.  measured  reliability  and  maintainabil- 
ity values  for  a similar  system  and  ten- 
tative reliability  and  maintainability 
requirements  for  the  new  system, 

3.  a defined  operational  life  profile  to  in- 
clude one  or  more  missile  profiles,  and 

4.  either  measured  environmental  stress 
data  from  similar  systems  or  guaran- 
teed access  to  such  data  as  input  to  the 
initial  design  process. 

The  Department  of  Defense  wants  re- 
sponding contractors  to  look  at  reliable,  main- 
tainable performance  in  field  service,  rather 
than  at  the  minimum  values  needed  to  pass  a 
series  of  stylized  laboratory  tests.  The  contrac- 
tors must  be  told  to  do  this  and  the  government 
must  provide  the  information  to  the  contractors 
to  enable  the  contractors  to  achieve  the  desired 
result.  The  contractors  must  be  assisted  as  well 
as  motivated  to  design  reliability  and  maintain- 
ability into  their  systems. 

Mathematical  Modeling  of  various  de- 
sign approaches,  to  include  parts  derat- 
ing and  system  packaging,  shall  be 
used  to  ensure  that  all  field  service  re- 
quirements, including  reliability  and 
maintainability  characteristics,  are  be- 
ing met.  Iteration  of  these  models  shall 
be  emphasized  as  a means  of  establish- 
ing a sound  design  approach;  however, 
the  quantitative  values  they  produce 
shall  not  be  used  as  measures  of  merit 
for  source  selection,  either  directly  or 
as  input  data  for  life  cycle  cost  esti- 
mates. 
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At  this  point  government  expertise  is 
needed  to  evaluate  how  responding  contractors 
propose  to  go  about  ensuring  that  Held  service 
reliability  and  maintainability  requirements  are 
met.  Several  comments  to  the  Draft  DODD 
SOOO.x  said  that  the  draft  directive  did  not  place 
sufficient  emphasis  on  designing  reliability  and 
maintainability  into  the  system.  This  is  where 
such  emphasis  belongs.  What  the  contractor 
proposes  to  do  about  stress  analysis,  parts  der- 
ating, worst  case  and  sneak  circuit  analysis,  fail- 
ure modes  and  effects  analysis,  and  “test-fix- 
test”  engineering  development  all  are  more  im- 
portant than  is  organization  of  reliability  and 
maintainability  staff  functions  or  how  high  a 
“predicted  mean-time-between-failure”  can  be 
produced.  Special  attention  is  needed  to  ascer- 
tain how  well  the  responding  contractors  have 
integrated  performance,  reliability  and  main- 
tainability and  environmental  considerations  in 
contractor  system  design  and  program  plans. 
Although  legal  staff  in  both  government  and  in- 
dustry want  and  need  some  kind  of  hard  criteria 
for  source  selection,  efforts  to  meet  that  need 
cannot  substitute  for  sound  engineering  judg- 
ment. How  engineering  judgment  can  be 
brought  to  bear  in  source  selection  decisions 
that  are  made  before  development,  before  test 
data  are  available,  and  under  prohibitions  on 
the  use  of  Military  Specifications  and  Standards 
is  a problem  larger  than  DODD  SOOO.x — and  it 
is  a major  reason  for  the  emphasis  on  continu- 
ing competition  into  engineering  development. 

Prototype  Demonstrations  shall  be 
used  to  refine  theoretical  reliability  and 
maintainability  predictions  and  vali- 
date the  proposed  design  approach. 
These  demonstrations  shall  test  per- 
formance, reliability  and  maintainabil- 
ity characteristics  and  environmental 
stresses  simultaneously.  Full  qualifica- 
tion tests  shall  not  be  required  during 
the  demonstration  and  validation 
phase  unless  the  program  bypasses  en- 
gineering development  and  goes  di- 
rectly into  production. 

Prototype  demonstrations  are  spot  tests  in- 
tended to  explore  and  resolve  particular  risks. 
It  is  important  that  the  spot  tests  not  be  frag- 
mented to  the  extent  that  a design  is  optimized 


for  performance  on  a laboratory  bench.  How- 
ever, sophisticated  “mission  profile”  qualifica- 
tion tests  are  not  appropriate  when  early  proto- 
types do  not  represent  the  design  approach  and 
equipment  configuration  intended  for 
production.  For  example,  it  makes  little  sense 
to  spend  much  time  and  money  on  full-blown 
qualification  tests  of  “chip  and  wire”  proto- 
types when  large  scale  integrated  circuits  will 
be  employed  in  the  production  unit.  Qualifica- 
tion tests  are  necessary  if  a decision  is  made  to 
produce  the  validation  phase  prototype. 

Trade  offs.  Tentative  reliability  an<J 
maintainability  requirements  shall  be 
subjected  to  continuing  review  for 
technical  feasibility  and  trade  offs  to 
balance  their  future  effects  on  opera- 
tional readiness,  effectiveness  and  own- 
ership cost  with  their  more  immediate 
effects  on  acquisition  cost  and  sched- 
ule. 

Such  trade  offs  are  the  Program  Manager’s  re- 
sponsibility. This  provision  gives  notice  that  re- 
liability and  maintainability  requirements  are 
not  sacred.  The  provision  also  indicates  that 
requirements  shall  not  be  eliminated  or  reduced 
to  ease  acquisition  cost  and  schedule  difficulties 
without  explaining  the  detrimental  effects  of 
such  reductions  on  the  field  service  cost- 
effectiveness  of  the  system.  Program  review  and 
decision  authorities  above  the  Program  Mana- 
ger are  alerted  to  the  potential  danger  of  “rob- 
bing Peter  to  pay  Paul”  when  reliability  and 
maintainability  requirements  are  reduced.  It  is 
the  responsibility  of  these  authorities  to  make 
sure  the  proposed  trade  offs  are  in  fact  life  cycle 
cost-effective.  That  is  about  as  far  as  a DOD  Di- 
rective can  go  in  attempting  to  resolve  what  has 
long  been  a major  problem  in  the  way  reliability 
and  maintainability  considerations  have  been 
addressed  during  system  acquisition. 

Enforceable  Reliability  and  Maintain- 
ability Requirements  shall  be 
presented  for  approval  at  Milestone  II 
and  included  as  quality  assurance 
provisions  in  the  specification  for  engi- 
neering development.  Each  reliability 
and  maintainability  characteristic  shall 
be  represented  by:  a mandatory  growth 


Vol  I,  No.  5 


139 


function  or  series  of  intermediate 
management  thresholds  for  develop- 
ment testing,  and  the  minimum  accept- 
able value  and  confidence  level  to  be 
demonstrated  prior  to  the  production 
decision.  Reliability  and  maintainabil- 
ity growth  shall  not  be  presented  as 
continuing  into  production  unless  con- 
current development  and  production 
has  been  specifically  authorized.  / 

Those  requirements  shown  in  Section  4, 
Quality  Assurance  Provisions,  of  an  equipment 
Specification  are  enforceable;  those  re- 
quirements shown  elsewhere  are  not.  The  same 
is  true  of  environmental  stresses  and  test 
conditions/procedures.  The  contractor  can  be 
told  to  design  for  “world-wide”  use,  but  in  fact 
he  has  to  design  for  the  criteria  in  Section  4 be- 
cause that  is  what  the  government  is  paying 
him  to  prove. 

The  purpose  of  engineering  development  is 
to  mature  the  approved  system  design.  Engi- 
neering development  is  the  place  for  “reliability 
growth” — and  maintainability  improvement. 
Effective  management  requires  that  reliability 
and  maintainability  growth  be  measured 
against  intermediate  points  in  a series  of  inter- 
mediate milestones  so  decisions  can  be  made 
early  to  modify  the  equipment.  Required 
growth  must  reach  a minimum  value  before  the 
system  is  evaluated,  and  the  system  must  be 
evaluated  prior  to  production.  Growth  does  not 
happen  automatically  nor  without  investment 
of  time  and  money.  If  growth  is  predicted  to 
continue  through  production  and  into  field  ser- 
vice, that  predicted  “growth  curve”  implies 
concurrent  development  and  production  and 
maturation  in  field  service — whether  or  not  the 
implication  is  made  clear.  Rather  than>.  allow 
the  implication  of  concurrence  to  go  unrecog- 
nized— which  may  well  make  it  a “primrose 
path”  for  the  Program  Manager  and  higher 
decision  authorities — DODD  5000.x  requires 
that  growth  requirements  be  made  explicit.  The 
Program  Manager  and  higher  decision  authori- 
ties then  are  free  to  decide  each  case  on  its  own 
merits. 

Reliability  Test  Duration.  The  maxi- 
mum test  time  for  demonstration  of 


compliance  with  reliability  require- 
ments prior  to  the  production  decision 
shall  be  established  prior  to  Milestone 
II  and  included  in  quality  assurance 
provisions  of  the  equipment  specifica- 
tion for  engineering  development.  Test 
time  and  number  of  test  articles  shall 
be  selected  to  balance:  (1)  total  test  cost 
and  impact  cost  of  any  addition  to 
length  of  program  schedule  owing  to 
reliability  testing,  (2)  the  cost  increase 
of  requiring  the  contractor  to  design 
more  reliable  equipment  or  to  fabricate 
a larger  number  of  samples  simply  to 
pass  a shorter  reliability  test,  and  (3) 
the  desired  degree  of  statistical  confi- 
dence or  decision  risk. 


This  is  a major  performance/cost/  sched- 
ule trade  off  that  is  now  obscured  in  the  selec- 
tion of  a Military  Standard  reliability  test  plan. 
The  trade  off  involves  a three-way  choice:  (1) 
increase  design  reliability  to  shorten  test  time 
and  number  of  samples,  (2)  add  test  time  and 
samples  to  reduce  pressure  on  system  design,  or 
(3)  reduce  statistical  confidence  (increase  “deci- 
sion risk”)  to  reduce  pressure  on  the  design,  to- 
tal test  time  and  number  of  samples.  The  Pro- 
gram Manager  and  the  contractor  need  to  know 
the  cost  of  statistical  confidence  or  decision  risk 
before  entering  into  a contract  for  Full-Scale 
Engineering  Development. 


FULL  SCALE  ENGINEERING 
DEVELOPMENT  PHASE 


In  this  phase  both  DOD  and  industry  need 
to  place  more  emphasis  on  reliability  and  main- 
tainability considerations  than  has  been  done  in 
the  past.  Although  reliability  and  maintainabil- 
ity have  to  be  “designed  into  the  system,”  a 
team  of  human  beings  cannot  anticipate  and 
eliminate  all  design  deficiencies.  The  choice  is 
to  find  the  remaining  deficiencies  and  eliminate 
them  during  engineering  development,  or  wait 
for  the  deficiencies  to  surface  in  early  field  ser- 
vice— where  every  design  change  takes  longer 
and  costs  more.  Research  and  Development 
(Test  and  Evaluation)  funds  are  not  plentiful. 


140 


Defense  Systems  MimgemenI  Review 


i 


Hence  DODD  5000.x  makes  a sharp  distinc- 
tion between  development  testing  that  is  in- 
tended to  improve  the  equipment  and  qualifica- 
tion tests  that  are  intended  to  determine  the 
compliance  of  a particular  configuration.  The 
directive  emphasizes  the  former  and  limits  the 
latter.  Also  the  directive  is  written  to  ensure 
that  improvements  actually  are  incorporated — 
because  if  the  improvements  are  not  made,  the 
business  of  engineering  development  is  a waste 
of  time  and  money. 

Development  Testing.  Equipment  used 
for  full-scale  engineering  development 
shall  be  operated  under  progressively 
more  realistic  test  conditions  and  pro- 
cedures to  mature  the  design  as  rapidly 
as  possible  by  disclosing  and  correcting 
latent  design  deficiencies. 

There  is  nothing  unusual  in  this  definition 
of  development  testing.  What  is  new  is  the  em- 
phasis on  finding  and  correcting  reliability  and 
maintainability  deficiencies  as  well  as  perfor- 
mance limitations.  The  DODD  5000.x  states 
that  Development  Test  and  Evaluation  at  the 
system  level  of  assembly  is  the  same  thing  as 
Test-Analyze-Find-Fix  (TAFF)  at  the  subsys- 
tem and  equipment  levels  of  assembly.  The  con- 
tractor should  conduct  Development  Test  and 
Evaluation  in  his  own  facilities,  using  his  most 
qualified  people  (especially  failure  analysts  and 
redesign  engineers),  under  minimum  con- 
straints and  with  minimum  administrative  de- 
lay. Since  this  is  the  only  kind  of  testing  specifi- 
cally intended  to  get  design  deficiencies  cor- 
rected, the  government  needs  to  ensure  this 
type  testing  is  not  squeezed  out  of  the  program. 
Stipulation  of  a minimum  as  well  as  a maxi- 
mum time  for  development  testing  is  one  way  to 
inform  the  contractor  the  government  is  serious 
about  maturing  the  design  before  producing  the 
item. 

Two  problems  are  typical  of  reliability  and 
maintainability  development  testing.  One  prob- 
lem is  that  a kind  of  ‘‘reliability  growth"  can  be 
obtained  just  by  replacing  "failed”  parts.  The 
replacement  amounts  to  screening  faulty  parts 
out  of  the  full-scale  development  models,  and 
the  apparent  reliability  growth  does  not  trans- 
fer to  production  units.  The  DODD  SOOO.x 


states  that  such  growth  indicates  the  need  for 
100  percent  preconditioning  (“Bum-In”)  of 
production  units.  The  other  problem  is  the  han- 
dling of  failures,  downtime,  maintenance 
manhours  and  materiel  costs  that  a proposed 
design  change  is  expected  to  eliminate.  As 
stated  in  DODD  SOOO.x  all  test  data  are  rele- 
vant to  field  service  unless  and  until  a design 
change  is  actually  incorporated  and  is  verified 
as  being  effective  in  reducing  reliability  and 
maintainability  problems.  The  old  reliability 
and  maintainability  values  then  become  test- 
peculiar  on  the  basis  that  they  pertain  to  a 
former  design  configuration.  Future  versions  of 
the  directive  will  have  to  place  emphasis  on  de- 
sign change  to  improve  maintainability  charac- 
teristics. Several  comments  pointed  out  that 
maintainability  is  “locked  in”  once  the  design  is 
frozen. 

Incorporation  of  Design  Changes. 
Cost-effective  reliability  and  maintain- 
ability design  changes  should  be  incor- 
porated and  verified  as  early  as  possi- 
ble during  Development  Test  and 
Evaluation/Test-Analyze-Find-Fix. 

Design  changes  shall  be  incorporated 
and  verified  to  provide  representative 
equipment  for  test  and  evaluation  prior 
to  the  production  decision.  Acquisition 
strategy  shall  ensure  that  sufficient 
time  and  resources  are  available  to  in- 
corporate cost-effective  reliability  and 
maintainability  improvements  at  this 
point  in  the  program. 

This  provision  is  self-explanatory.  All  too 
often  when  deficiencies  are  found,  there  is  an 
inability  to  correct  them  because  cost  and 
schedule  planning  are  too  success-oriented. 
Granted  that  engineering  development  is 
funded  with  a different  color  of  money  than 
production,  and  that  less  funds  are  available 
from  Research  and  Development  (Test  and 
Evaluation)  than  from  procurement  accounts, 
the  high  retum-on-investment  for  funding  little 
“fixes”  with  big  payoffs  in  terms  of  field  service 
reliability  and  maintainability  make  this  provi- 
sion necessary. 
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Test  and  Evaluation  Prior  to  the 
Production  Decision.  Final  preproduc- 
tion prototypes  shall  be  tested  under 
the  most  realistic  simulation  of  field 
service  available  to  provide  demon- 
strated reliability  and  maintainability 
values  for  evaluation  prior  to  Mile- 
stone III. 

Test  and  evaluation  prior  to  the 
production  decision  has  been  a major  aspect  of 
DOD  acquisition  policy  for  the  past  7 years — 
since  former  Deputy  Secretary  of  Defense 
David  Packard  signed  his  memorandum  on  sys- 
tem acquisition  in  May,  1970 — but  it  is  only 
now  beginning  to  be  reflected  in  Military  Stan- 
dards on  reliability  and  maintainability.  The 
DODD  5000.x  requires  that  performance,  reli- 
ability and  maintainability  and  environmental 
qualification  tests  be  integrated  insofar  as  prac- 
tical and  conducted  prior  to  production.  The 
qualification  of  “first  production  units”  is  not 
an  acceptable  substitute  for  the  testing  of  final 
preproduction  units.  Preproduction  qualifica- 
tion tests  of  subsystems  and  equipment  are  the 
same  as  Initial  Operational  Te<!  and  Evaluation 
(lOT&E)  of  systems.  Both  of  these  kinds  of  test 
should  be; 

• short  to  avoid  the  impact  cost  of  pro- 
gram delay, 

• realistic  to  project  field  service  reliabil- 
ity and  maintainability  values  based  on 
test  results  as  inputs  to  the  production 
decision,  and 

• independent  of  the  developer  to  avoid 
the  inherent  conflict  of  interest  in- 
volved in  having  the  contractor  test 
and  evaluate  his  own  product. 

Several  comments  pointed  out  the  need  for 
clarification  regarding  maintainability  demon- 
strations at  the  system  level  of  assembly.  The 
demonstrations  are  different  from  the  collec- 
tion of  maintainability  data  from  an  integrated 
test  of  subsystems  or  equipment  because 
demonstrations  can  and  should  be  accelerated 
by  the  introduction  of  simulated  failures  (as  op- 
posed to  waiting  for  failures  to  occur). 


Maintainability  demonstrations  must  be  con- 
ducted by  personnel  having  the  skill  level,  tools, 
test  equipment  and  technical  publications  spec- 
ified for  field  service.  The  use  of  highly  quali- 
fied scientists  and  engineers  simply  invalidates 
the  demonstration  and  makes  the  results  opti- 
mistic. Some  comments  said  it  is  impossible  for 
the  contractor  to  have  tools,  test  equipment  and 
technical  publications  ready  prior  to  the 
production  decision.  If  so,  it  is  questionable 
whether  the  system — being  a combination  of 
hardware,  software  and  people — is  ready  for 
production  and  deployment.  Again,  if  concur- 
rent development,  production  and  deployment 
are  the  fact,  that  fact  should  be  made  explicit 
and  judged  on  its  merits,  rather  than  obscured 
in  a reliability  and  maintainability  program. 

The  DODD  5000.x  stated  that  the  reliabil- 
ity aspects  of  a qualification  test  need  not  be 
prolonged  beyond  the  point  at  which  minimum 
acceptable  values  have  been  demonstrated  at 
minimum  acceptable  confidence  levels.  Accept- 
ing equipment  on  the  basis  of  demonstrated 
confidence  levels  increases  the  “consumer’s  de- 
cision risk,”  that  is,  the  probability  of  accepting 
less  than  the  minimum  acceptable  value.  How- 
ever, DODD  5000.x  did  not  say  that  the  Pro- 
gram Manager  has  to  stop  the  test  and  accept 
the  equipment  on  the  basis  of  demonstrated 
confidence  levels.  The  directive  said  that  such  a 
decision  is  within  the  authority  of  the  Program 
Manager  if  he  chooses  to  exercise  this  preroga- 
tive. Such  a decision  is  appropriate  when  the 
Program  Manager  has  selected  a fixed-length 
reliability  test  plan  and  the  equipment  demon- 
strates significantly  more  than  minimum  ac- 
ceptable reliability.  While  this  appears  to  be  a 
highly  technical  detail  of  statistical  theory,  it  is 
in  fact  a performance/cost/  schedule  trade  off 
that  Program  Managers  need  to  be  aware  they 
can  make.  In  some  cases,  sacrificing  the  last  10 
percent  of  “consumer’s  decision  risk,”  without 
sacrificing  “demonstrated  confidence  level” 
can  cut  the  time  required  for  a fixed-length  test 
by  85  percent. 

To  ensure  that  decision  authorities  know 
what  the  reliability  and  maintainability  values 
shown  at  Milestone  III  really  mean,  DODD 
5000.x  states  that  demonstrated  test  results  will 
be: 


142 


Defense  Systems  Management  Review 


• measured  within  a specified  confidence 
interval, 

• adjusted  to  remove  any  difference  be- 
tween field  service  and  contractual 
definitions, 

• adjust  for  degree  of  test  realism,  and 

• compiled  to  project  field  service  relia- 
bility and  maintainability  values  at  the 
sytem  level  of  assembly. 

This  is  the  “achievements”  side  of  the  “require- 
ments/achievements audit  trail.”  The  entire  au- 
dit trail  does  not  have  to  be  presented  at  Mile- 
stone III  reviews,  but  decision  authorities  have 
to  know  it  has  been  accomplished  and  the  audit 
trail  must  be  available  on  request  to  substanti- 
ate the  reliability  and  maintainability  values 
presented  as  achievements. 

Enforceable  reliability  and  maintainability 
requirements  for  the  production  and  deploy- 
ment phase  must  be  presented  at  Milestone  III. 
These  too,  are  field  service  reliability  and  main- 
tainability values.  If  these  values  are  higher 
than  those  demonstrated  by  Initial  Operational 
Test  and  Evaluation  and  preproduction  qualifi- 
cation tests,  the  Program  Manager  should  be 
prepared  to  explain  exactly  how  that  improve- 
ment will  be  obtained  and  who  will  pay  for  it.  A 
theoretical  “growth  curve”  is  not  acceptable  at 
this  point  unless  concurrent  development, 
production  and  maturation  in  field  service  is 
part  of  the  production  decision.  Even  then,  the 
growth  has  to  be  planned,  funded  and  managed, 
or  it  cannot  be  expected  to  occur. 

Once  approved  at  Milestone  III,  field  ser- 
vice reliability  and  maintainability  require- 
ments in  the  Decision  Coordinating  Paper  or 
equivalent  DOD  documentation  must  be  trans- 
lated into  contractual  reliability  and  maintaina- 
bility requirements  and  included  in  Section  4 of 
the  equipment  Specification  for  the  production 
phase.  That  makes  the  requirements  enforce- 
able criteria  of  contractual  compliance,  to  be 
evaluated  by  production  acceptance  tests. 


PRODUCTION  AND 
DEPLOYMENT  PHASE 

Again,  DODD  5000.x  emphasizes  those 
tests  intended  to  improve  the  equipment  and 
de-emphasizes  those  intended  to  prove  or  dis- 
prove compliance,  in  terms  of  the  amount  of 
time  and  money  spent  on  the  tests.  The  direc- 
tive calls  for  100  percent  inspection  and 
strongly  recommends  100  percent  precondi- 
tioning (“Bum-In”)  to  reduce  infant  mortality 
in  field  service.  Further,  the  directive  calls  for 
Integrated  (Mission  Profile)  Production  Accep- 
tance Tests,  with  real  “teeth”  in  terms  of  con- 
tractual compliance,  but  the  directive  limits  the 
tests  to  a sampling  in  the  interests  of  cost  and 
schedule  reduction. 


Preconditioning  (“Burn-In”).  All  de- 
liverable end  items  shall  be  submitted 
to  a short  test,  at  the  highest  practical 
level  of  assembly,  to  disclose  weak 
parts  and  manufacturing  defects  for 
correction  prior  to  delivery. 

There  were  many  potentially  useful  com- 
ments in  regard  to  this  provision.  The  differ- 
ence between  “Inspection,”  “Parts  Screening,” 
“Bum-In,”  and  “Preconditioning”  was  dis- 
cussed. Several  comments  highlighted  the  need 
for  management  flexibility  with  regard  to  how 
preconditioning  is  done  and  how  it  can  be 
changed  during  the  course  of  the  production 
mn.  None  of  the  comments  said  precondition- 
ing is  not  needed  nor  that  it  does  not  more  than 
pay  for  itself — for  both  the  contractor  and  the 
government.  The  statement  in  DODD  5000.x 
to  the  effect  that  environmental  stresses  for  pre- 
conditioning should  be  selected  to  disclose  de- 
fects, rather  than  to  simulate  the  operational 
environment,  was  well  received.  So  was  the 
provision  that  these  tests  must  be  done  by  the 
contractor.  However,  there  was  some  dis- 
agreement as  to  whether  preconditioning  tests 
should  be  managed  against  “accept/reject  crite- 
ria.” The  position  taken  in  the  draft  Directive 
was  that  it  is  unwise  to  penalize  the  contractor 
for  finding  and  fixing  defects,  since  that  is  ex- 
actly what  the  government  wants  the  contrac- 
tor to  do. 
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Production  Acceptance  Tests.  After 
items  have  completed  the  precondi- 
tioning test  and  have  been  certified 
ready  for  delivery,  a few  samples  shall 
be  selected  at  random  from  each 
production  lot  and  resubmitted  to  the 
preproduction  qualification  test. 

Production  acceptance  test  conditions  and 
procedures  must  be  identical,  within  specified 
tolerances,  to  those  used  for  preproduction 
qualification  tests  of  the  same  equipment.  A 
more  stringent  test  is  not  fair  to  the  contractor. 
A less  stringent  test  allows  the  contractor  to 
shave  comers  after  the  government  is  commit- 
ted to  produce  the  design. 

Contractual  requirements  for  production 
acceptance  tests  must  be  enforceable.  That  is, 
“accept”  must  mean  “compliant,”  and  “reject” 
must  mean  “noncompliant,”  not  “take  correc- 
tive action  and  retest.”  (The  automatic  “reject 
and  retest”  feature  of  MIL-STD-781B  has  been 
removed  from  MIL-STD-781C  because  this  fea- 
ture meant  that,  in  practice,  the  only  way  to 
stop  a reliability  test  was  to  accept  the  equip- 
ment— one  way  or  another.  Any  similar  fea- 
tures in  as-yet-unrevised  Military  Standards 
must  be  waivered  as  a matter  of  policy.) 

All-equipment  production  acceptance 
tests  are  discouraged,  but  not  prohibited.  These 
tests  are  far  more  expensive  than  precondi- 
tioning tests,  because  of  the  need  for  realistic 
“Mission  Profile”  conditions  and  procedures. 
All-equipment  production  acceptance  tests 
were  not  prohibited  because  there  are  a few  cas- 
es— such  as  in  man-rated  spacecraft — where 
such  testing  can  be  justified. 

Like  preproduction  qualification  tests  and 
initial  operational  test  and  evaluation, 
production  acceptance  tests  should  be  indepen- 
dent of  the  producing  contractor.  Once  a 
product  is  on  the  shipping  dock  and  certified 
ready  for  delivery,  the  contractor  should  be  pre- 
pared to  have  the  product  delivered  to  field  ser- 
vice or  diverted  to  an  independent  laboratory 
that  will  test  the  product  under  specified  con- 
ditions and  proc^ures  designed  to  simulate 
field  service. 


Field  Evaluation  of  early  deployed  sys- 
tems shall  be  the  primary  method  of 
determining  compliance  with  field  ser- 
vice reliability  and  maintainability 
thresholds  approved  at  Milestone  III. 
Program  plans  for  production  and  de- 
ployment shall  contain  provisions  for: 
(a)  feedback  of  measured  reliability 
and  maintainability  values  to  the  Pro- 
gram Office  and  the  contractor(s),  (b) 
determination  of  compliance,  and  (c) 
rapid  correction  of  residual  deficien- 
cies and  defects. 

The  field  evaluation  provision  is  neither 
new  nor  exotic.  The  Congress  evaluates 
achievement  of  reliability  and  maintainability 
requirements  in  the  Decision  Coordinating  Pa- 
per against  reliability  and  maintainability  val- 
ues collected  by  the  maintenance  data  reporting 
system  used  in  field  service.  A problem  is  gener- 
ated when  the  definitions  differ.  Contractors 
keep  asking  for  feedback  of  field  service  reliabil- 
ity and  maintainability  data.  There  is  no  appar- 
ent reason  why  the  contractors  should  not  have 
this  information  as  a matter  of  course  from  sys- 
tems they  have  produced,  or  on  request  from 
other  systems.  Several  systems  have  been 
fielded  recently  with  what  amounted  to  a guar- 
antee, by  the  developing  command  to  the  oper- 
ating command,  that  the  system  would  meet 
Milestone  III  reliability  and  maintainability  re- 
quirements or  be  corrected  by  the  developing 
command.  That  kind  of  policy  is  worthy  of  en- 
couragement and  support  as  a matter  of  DOD 
policy. 

Field  Service  Warranties.  The 
production  and  deployment  phase  may 
contain  provisions  for  either  develop- 
ing agency  or  contractor  support  of  de- 
ployed systems.  These  provisions  shall 
distinguish  between  maintenance  sup- 
port and  further  reliability  and  main- 
tainability improvement. 

This  distinction  is  necessary  to  prevent 
support  and  improvement  provisions  from 
working  against  each  other.  Maintenance  sup- 
port contracts  must  be  defined  with  care  and 
with  regard  to  who  is  responsible  for  various 


144 


Defense  Systems  Management  Review 


kinds  of  failures,  downtime  hours,  maintenance 
manhours  and  materiel  costs.  Improvement 
contracts  need  broad  definition  to  allow  and  en- 
courage the  contractor  to  identify  needed  im- 
provements in  hardware,  software,  training, 
manning,  organization,  etc.  Where  these  two 
concepts  have  been  mixed  in  the  same  contract, 
the  results  are  expectable:  the  need  for  improve- 
ment will  be  found  in  an  area  not  covered  by  the 
maintenance  contract  and  the  provisions  of  the 
maintenance  contract  will  not  be  enforceable. 

RESPONSIBILITIES 

Effective  management  demands  clear  defi- 
nition of  areas  of  responsibility,  with  commen- 
surate authority.  Implementation  of  this  princi- 
ple demands  an  answer  to  the  question,  “Who 
is  involved  in  this  activity”?  Where  the  subject 
is  the  reliability  and  maintainability  of  defense 
systems  and  equipment,  the  answer  must  in- 
clude mention  of  the  operating  commands  who 
write  initial  requirements,  the  developing  com- 
mands who  meet  those  requirements,  and  the 
supporting  commands  who  inherit  the  final 
product.  Effective  management  must  include 
the  Program  Managers  procuring  activities, 
contractors  and  subcon  ractors.  Management 
must  include  the  program  review  and  decision 
authorities  in  the  chain  of  command  above  any 
given  Program  Manager  and  the  independent 
review  agencies  that  support  those  authorities. 
Effective  management  must  include,  but  can  no 
longer  be  limited  to,  the  professional  reliability 
and  maintainability  staff  functions  in  all  of 
these  agencies.  Given  that  a DOD  Directive 
sets  policy  but  does  not  tell  the  DOD  Compo- 
nents or  the  contractors  how  to  implement 
those  policies,  or  how  to  organize,  DODD 
5000.x  has  addressed  reliability  and  maintain- 
ability as  a command  responsibility. 

PROGRAM  MANAGER 

The  designated  Program  Manager  or  pro- 
curing activity  is  responsible  for  the  field  ser- 
vice reliability  and  maintainability  charac- 
teristics of  both  Contractor  Furnished  Equip- 
ment and  Government  Furnished  Equipment. 
The  Program  Manager  is  responsible  for  selec- 
tively tailoring  reliability  and  maintainability 


tasks,  integrating  these  tasks  into  each  phase  of 
the  acquisition  program,  and  ensuring  that  the 
tasks  are  satisfactorily  accomplished.  Too,  the 
Program  Manager  is  responsible  for  the  transla- 
tion of  field  service  reliability  and  maintainabil- 
ity requirements  into  contractual  requirements, 
and  for  the  translation  of  contractual  reliability 
and  maintainability  achievements  into 
projected  field  service  reliability  and  maintaina- 
bility values.  Further,  the  Program  Manager  is 
responsible  for  the  inclusion  of  quantitative  re- 
liability and  maintainability  values  as  an  inte- 
gral part  of  performance/cost/schedule  and  life 
cycle  cost  trade  offs  within  established  thresh- 
olds. Portions  of  specified  authority  may  be  del- 
egated to  the  contractor  or  to  outside  agencies, 
but  the  Program  Manager  retains  overall  re- 
sponsibility and  authority  to  manage  the  relia- 
bility and  maintainability  aspects  of  the  pro- 
gram. 

INDEPENDENT  REVIEW  AGENCIES 

The  independent  test  and  evaluation  agen- 
cies in  each  DOD  Component  are  responsible 
for  reviewing  reliability  and  maintainability 
tests  and  test  results  in  accordance  with  the 
provisions  of  DODD  5000.3,  “Test  and  Evalua- 
tion.” The  Cost  Analysis  Improvement  Group 
is  responsible  for  reviewing  the  acquisition  and 
ownership  cost  impact  of  reliability  and  main- 
tainability requirements  and  achievements  in 
accordance  with  the  provisions  of  DODD 
5000.4,  “OSD  Cost  Analysis  Improvement 
Group.”  Organizations  having  equivalent  func- 
tions will  perform  these  reviews  for  less-than- 
major  system  acquisition  programs  managed 
within  the  DOD  Components. 

Several  comments  to  the  draft  DODD 
5000.x  recommended  that  an  Independent  Re- 
liability and  Maintainability  Review  Authority, 
similar  to  the  Independent  Test  and  Evaluation 
Agencies  and  Cost  Analysis  Improvement 
Groups,  be  established  in  each  of  the  DOD 
Components.  Such  recommendations  were 
found  to  conflict  with  established  OMB  and 
DOD  policies  with  regard  to  management  lay- 
ering and  restriction  of  the  Program  Manager’s 
authority.  It  was  felt  that  another  review  would 
add  administrative  delays  to  the  acquisition 
schedule,  and  that  establishing  reliability  and 
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maintainability  as  a separate  organization 
would  not  serve  to  integrate  reliability  and 
maintainability  considerations  into  the  main- 
stream of  program  management.  Future  ver- 
sions of  DODD  5000.x  will  continue  to  treat  re- 
liability and  maintainability  as  system  perfor- 
mance and  design  characteristics.  This  ap- 
proach allows  the  DOD  Components  to  staff 
reliability  and  maintainability  functions,  at  all 
levels  of  organization,  as  necessary  and  appro- 
priate to  comply  with  the  substantive  provi- 
sions of  the  Directive. 

LINE  AUTHORITIES 

The  line  authorities,  in  the  chain  of  com- 
mand above  a given  Program  Manager  or  pro- 
curing activity,  are  responsible  for  the  re- 
liability and  maintainability  thresholds  they  ap- 
prove. The  line  authorities  shall  ensure  that: 


these  measurable  characteristics  are  key,  con- 
trolling factors  of  operational  readiness,  effec- 
tiveness and  life  cycle  cost,  a greater  degree  of 
management  attention  can  be  expected. 

MILITARY  DEPARTMENTS 

The  Milicary  Departments  are  responsible 
for: 

• establishing  realistic  reliability  and 
maintainability  requirements  for  all  ac- 
quisition programs  by  ensuring  ade- 
quate coordination  among  operating, 
developing  and  supporting  commands, 

• providing  measured  field  service  relia- 
bility and  maintainability  data  and  en- 
vironmental stress  data  to  the  Program 
Offices  and  contractors. 


• reliability  and  maintainability  thresh- 
olds are  realistic, 

• reliability  and  maintainability  values 
used  as  decision  information  are  rele- 
vant to  field  service  and  are  based  on 
demonstrated  test  results, 

• projected  field  service  reliability  and 
maintainability  values  satisfy  opera- 
tional readiness,  effectiveness  and  own- 
ership cost  requirements  at  affordable 
acquisition  cost  and  schedule,  and 

• the  Program  Manager  or  procuring  ac- 
tivity is  provided  with  sufficient  re- 
sources to  enable  compliance  with  the 
provisions  of  this  DODD  5000.  x. 

The  degree  to  which  reliability  and  main- 
tainability considerations  become  an  integral 
part  of  defense  system  acquisition  depends 
upon  the  degree  of  management  attention  re- 
ceived from  program  review  and  decision  au- 
thorities. Reliability  and  maintainability  staff 
who  report  to  these  authorities  must  ensure 
that  all  reliability  and  maintainability  consider- 
ations are  clear  and  presented  in  a form  amena- 
ble to  high  level  decisionmaking.  Once  reliabil- 
ity and  maintainability  characteristics  are  prop- 
erly defined,  and  line  authorities  know  that 


• maintaining  an  adequate  reliability  and 
maintainability  technology  base  and 
corporate  memory  in  support  of 
present  and  future  acquisition  pro- 
grams, and 

• revising  Military  Regulations,  Specifi- 
cations and  Standards  as  necessary  to 
comply  with  the  provisions  of  DODD 
5000.x. 

Realistic  reliability  and  maintainability  re- 
quirements must  be  based  on  experience  rather 
than  on  theoretical  analysis  of  the  reliability 
and  maintainability  values  desired.  The  operat- 
ing commands  now  write  initial  requirements 
for  new  systems  and  equipment.  The  sup- 
porting commands  will  have  to  become  more 
involved  in  writing  the  maintenance  reliability 
and  maintainability  [wrtions  of  those  re- 
quirements. Effort  will  be  required  to  make  use 
of  existing  reliability  and  maintainability  and 
environmental  stress  data,  and  to  fill  the 
present  gaps  where  data  is  lacking.  The  alterna- 
tive is  continued  reliance  on  stylized  Military 
Standards  that  do  not  deliver  satisfactory  relia- 
bility and  maintainability  in  field  service.  New 
technology  will  be  need^  to  actually  improve 
reliability,  and  that  means  greater  attention  to 
reliability  on  the  part  of  both  government  and 
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i industry  laboratories.  Major  efforts  will  be  re- 

quired to  upgrade  present  Military  Regula- 
tions, Specifications  and  Standards,  many  of 
which  were  written  to  support  total  package 
' procurement  and  have  not  been  subjected  to 

f needed  revision.  All  of  these  activities  must  be 

[ supported,  funded  and  managed  by  the  com- 

mand line  in  each  DOD  Component  if  im- 
proved readiness  and  effectiveness,  with  re- 
duced life  cycle  cost,  is  to  be  attained. 

OFFICE  OF  THE  SECRETARY 
OF  DEFENSE 

The  Acquisition  Executive  has  prime  re- 
sponsibility for  DODD  5000.  X and  for  its  im- 
j plementation.  The  Assistant  Secretary  of  De- 

; fense  (Manpower,  Reserve  Affairs  and  Logis- 

tics) and  the  Director  (Test  and  Evaluation) 
will  participate  in  review  of  implementing  doc- 
uments. 

f 

t 
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CONCLUSION 

Like  the  weather,  everybody  talks  about 
reliability  and  maintainability  problems  but  lit- 
tle seems  to  get  done  about  them.  Like  the 
weather,  no  possible  solution  is  going  to  please 
everybody.  The  farmers  pray  for  rain  while  the 
golfers  pray  for  sunshine.  Nevertheless,  the 
magnitude  of  reliability  and  maintainability 
problems  in  deployed  systems  and  equipment, 
and  the  ever-increasing  squeeze  on  both  acquis- 
ition and  ownership  cost,  indicate  the  time  has 
come  to  suspend  debate  and  move  out.  While  it 
is  a long  way  from  perfect  and  a long  way  from 
comprehensive,  DODD  5000.  x is  a major  step 
in  the  direction  of  reliable,  maintainable  sys- 
tems and  equipment  at  lower  life  cycle  cost.  On 
that  basis,  and  on  that  basis  only,  the  present 
and  future  revisions  of  DODD  5000.x  should 
be  reviewed  and  evaluated. 
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B-1  STRUCTURAL  TEST  PROGRAM 


by 

John  W.  Rustenburg,  Department  of  Air  Force 
Mark  A.  Owen,  Department  of  Air  Force 
William  L.  Geese,  Department  of  Air  Force 


The  B-1  Bomber  Program  began  in  1970  during  a time  when  “fly-before-buy”  was  driving 
aircraft  system  procurement  into  long  periods  of  development  prior  to  production  commitment. 

Theoretically,  subsystems  may  be  replaced  many  times  during  an  aircraft’s  useful  life,  but 
basic  structure  is  expected  to  continue  to  function  as  designed.  Careful  attention  was  therefore 
given  in  the  B-1  Program  to  disciplined  detail  structural  design  and  to  extensive  ground  and 
flight  testing. 

The  B-1  full  scale  structural  test  program  contained  many  separate  elements  that  may  be 
considered  under  the  broad  headings  of  static,  fatigue  and  flight  testing.  This  separation  into 
testing  regimes  was  not  intended  to  imply  independence.  The  results  from  one  regime  may  be 
critical  inputs  to  another.  The  thrust  of  the  testing  was  toward  flnding  and  eliminating  potential 
structural  problems  in  one  phase  of  testing  before  moving  on  to  the  next.  The  intent  was  to 
arrive  at  an  airframe  whose  structural  integrity  is  fully  understood  and  one  that  can  be  built  in 
production  quantities  with  high  confidence. 


STRUCTURAL  INTEGRITY 

The  results  from  testing  are  used  to  opti- 
mize system  design  as  well  as  to  aid  in  the 
production  decision  making  process. 

One  aspect  of  B- 1 development  receiving 
major  attention  was  structural  integrity.  The  F- 
1 1 1 and  C-5A  recovery  programs  had  demon- 
strated the  need  for  a unified  approach  to  struc- 
tural integrity  during  the  development  phase. 
Aircraft  structural  considerations  are  of  basic 
importance  since  the  structure  forms  a founda- 
tion for  the  remainder  of  the  weapon  system. 

Figure  1 is  a schematic  representative  of 
the  various  elements  of  the  B-1  structural  test 
program. 

STATIC  TEST  PROGRAM 

The  purpose  of  static  testing  on  the  B-1 
was  to  verify  analytically  predicted  margins  of 


safety  to  clear  the  way  for  later  flight  tests.  (As 
used  here,  “static”  refers  to  discrete  properties 
of  as-designed  structural  assemblies,  including 
strength,  stiffness  and  natural  frequency.)  The 
elements  of  the  B-1  static  test  program  included 
design  verification  and  ground  vibration  testing 
prior  to  the  first  flight  of  Aircraft  1 and  a struc- 
tural ground  test  program  prior  to  flight  of  Air- 
craft 2. 

In  the  past  an  airplane’s  static  capabilities 
could  only  be  evaluated  by  loading  the  full  air- 
frame to  design  load  levels  plus  a factor  of 
safety,  and  very  often  to  destruction.  Over  the 
years  airframe  static  design  practice  has  ad- 
vanced to  the  use  of  sophisticated  finite  element 
computer  programs.  As  finite  element  analysis 
techniques  developed,  math  modeling  im- 
proved. As  the  accuracy  of  these  methods  was 
demonstrated,  static  testing  innovations  natu- 
rally occurred.  If  complex  pieces  of  structure 
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can  be  designed  and  analyzed  using  these  finite 
element  techniques,  and  then  be  verified  by 
tests,  a more  simple  structure  designed  and  ana- 
lyzed using  the  same  methods  will  generally  be 
f adequate.  The  expense  of  test  verification  of  the 

I simpler  structure  therefore  need  not  be  borne. 


In  the  case  of  the  B-l,  original  plans  called 
for  a full  airplane  test  to  destruction  in  addition 
to  extensive  structural  configuration  develop- 
ment tests,  design  verification  tests  and  pre- 
flight operational  and  proof  loads  tests.  Because 
of  the  expense  of  this  program,  an  innovative 
look  was  taken  at  the  testing  requirements  to 
determine  the  design,  analysis  and  testing  abso- 
lutely necessary  and  realistic.  As  a result  the 
full  airplane  test  to  destruction  was  eliminated. 
Significant  program  dollars  were  saved  with  al- 
most no  additional  risk  to  the  B- 1 structural  in- 
tegrity. This  decision  allowed  the  earliest  possi- 
ble testing  of  the  most  critical  structural  assem- 
blies in  the  design  verification  test  program. 


i 


Design  Verification  Testing 


The  purpose  of  static  design  verification 
testing  was  to  validate  the  predicted  structural 
strength  margins  of  safety  prior  to  flight  of  any 
B-l  aircraft.  The  test  articles  consisted  of  com- 
ponents and  assemblies  representing  the  most 
critical  areas  of  the  airframe.  The  articles  in- 
cluded all  splices,  joints  and  fittings  appropriate 
to  the  specific  test  item  with  realistic  boundary 
condition  restraints.  Selection  of  the  test  com- 
ponents was  based  on: 

• Importance  of  the  component’s  struc- 
tural stiffness,  integrity,  and  function 
in  the  vehicle, 

• Degree  of  structural  redundancy  and 
complexity,  and 


[ • Method  of  construction  and  materiels 

I used  in  the  component  as  compared  to 

[ current  technology. 

i 

I Figure  2 shows  the  location  of  these  design  veri- 

fication test/articles  within  the  B-l  airframe. 


i 


The  static  design  verification  test  program 
on  the  major  assemblies  required  24  months  to 
complete.  The  program  met  the  requirement  of 
verifying  strength  margins  prior  to  flight  of  Air- 
craft 1.  In  the  course  of  testing,  design  loads 
were  altered  and  updated,  substructures  that 
had  failed  were  redesigned  and  replaced,  test 
conditions  were  rerun  and  various  other 
changes  were  made  to  the  program.  Finally,  the 
major  wing  carry  through  article  was  subject  to 
twelve  different  design  load  conditions  (six  to 
limit  load,  the  maximum  load  that  any  B-l 
would  be  expected  to  experience  in  service,  and 
six  to  ultimate,  150  percent  of  limit)  in  order  to 
thoroughly  test  all  portions  of  the  structures. 
Catastrophic  failure  was  finally  induced  at  1 1 1 
percent  of  the  wing  design  ultimate  load. 

A number  of  structural  failures  occurred 
during  testing  that  contributed  substantially  to 
the  time  required  to  complete  the  program. 
These  were  failures  for  which  remedies  could  be 
made  on  the  flight  articles.  Almost  all  failures 
occurred  in  various  parts  of  the  fuselage  struc- 
ture as  a result  of  one  or  both  of  the  following: 

• Poor  detail  design  in  corner  attach- 
ment locations  and  in  similar  areas  of 
abrupt  load  changes  resulting  from 
discontinuous  structure;  such  design 
deficiencies  compounded  an  already 
severe  load  redistribution  problem. 

• The  use  of  undersize  (shear  critical)  at- 
tachments for  joining  heavily  loaded 
members;  such  fasteners  were  unable 
to  take  advantage  of  the  substantial 
strength  of  the  basic  structure  being 
joined. 

The  airplane  fix  involved  simply  increasing  the 
size  and  material  strength  of  the  fasteners  and 
adding  a few  large  attachments  at  the  critical 
corner  locations.  This  modification  often  re- 
sulted in  doubling  the  allowable  joint  strength. 

An  exception  to  this  “easy  fix”  solution 
was  a bulkhead  failure  in  the  main  landing  gear 
wheel  well  region  of  the  aft  intermediate  fuse- 
lage where  an  unexpected  load  redistribution 
occurred.  Design  measures  involving  various 
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doublers,  stifTeners  and  increased  attachment 
strengths  were  tried  with  little  success.  The  alu- 
minum web  of  the  bulkhead  was  finally  re- 
placed with  a heavy  steel  member  of  excep- 
tional structural  integrity.  A similar  repair  was 
used  on  the  flight  test  aircraft  and  demon- 
strated on  the  design  verification  test  article. 

The  static  design  verification  test  articles 
were  a tool  through  which  design  strength 
could  be  checked,  inadequacies  uncovered  and 
redesigns  made.  The  tests  further  served  as  a ve- 
hicle for  testing  repairs  before  part  installation 
in  a flight  article.  When  a generic  problem  was 
found  with  implications  to  structure  not  being 
tested  (such  as  the  undersize  fasteners  men- 
tioned earlier),  similar  changes  were  made 
throughout  the  airframe.  The  result  was  high 
confidence  in  the  structural  strength  of  the 
flight  articles  as  they  entered  a very  demanding 
flight  test  program. 

Ground  Vibration  Testing 

The  purpose  of  the  ground  vibration  test  is 
to  determine  the  natural  vibration  modes  and 
frequencies  of  the  complete  airplane.  Ground 
vibration  test  constitutes  a proof  test  of  primary 
importance  in  determining  structural  flight  lim- 
its with  respect  to  flutter  as  well  as  other  areas 
where  vibration  modes  are  important,  such  as 
the  flight  control  surfaces,  nose  boom,  over- 
wing fairings  and  the  landing  gear. 

The  ground  vibration  test  was  conducted 
on  Aircraft  1 prior  to  first  flight  over  a nonstop, 
12  and  1/2  day  period.  The  entire  airplane  was 
mounted  on  a soft  suspension  system  with  at- 
tached vibration  shakers  and  tested  in  six  con- 
figurations including  four  different  wing  sweep 
angles  and  simulated  single  and  double  hydrau- 
lic failure  modes  of  the  horizontal  tail.  Symme- 
tric and  antisymmetric  frequency  sweeps  were 
made  for  each  of  the  six  configurations  re- 
sulting in  a total  of  twelve  separate  test  se- 
quences. 

The  vibration  modes  obtained  in  the 
ground  vibration  test  were  compared  with  ana- 
lytically predicted  modes  to  assess  the  stiffness 
of  the  actual  flight  article.  Also  the  vibration 


modes  were  used  in  calculations  to  determine 
“corrected”  flight  limits  and  enable  further 
comparison  with  similar  measurements  on 
wind  tunnel  flutter  models  to  establish  the  air- 
plane flutter  limits.  This  information  provided 
the  necessary  confidence  that  the  airplane 
would  exhibit  adequate  flutter  speed  margins  at 
the  design  flight  limits  and  that  flight  flutter 
testing  on  Aircraft  1 would  be  initiated  safely. 

Aircraft  2 Structural  Ground  Testing 

The  structural  ground  test  program  on 
Aircraft  2 included  the  limit  load  proof  test  in- 
tended to  demonstrate,  in  conjunction  with  the 
design  verification  tests,  the  static  strength 
structural  integrity  of  the  aggregate  airframe. 
Although  this  was  the  primary  purpose,  three 
additional  objectives  were  satisfied,  a structural 
deflection  test,  a limit  load  rigidity  test  and  a 
strain  gage  calibration  test.  Total  testing  ex 
tended  over  a period  of  approximately  6 
months. 

Limit  Load  Proof  Tests 

For  all  the  structural  ground  tests.  Air- 
craft 2 was  placed  within  a load  reaction  frame 
and  mounted  on  main  and  nose  landing  gear 
fixtures.  Nacelle  attach  point  beam  fixtures 
were  used  to  transmit  simulated  engine  and  na- 
celle inertial  loads.  Weapon  fitting  loads  were 
applied  through  simulated  weapons  launcher 
spools,  that  also  reacted  fuselage  x-axis  inertial 
loads.  Wing,  empennage  and  fuselage  loadings 
were  induced  through  tension,  shear  and  point 
loading  fixtures. 

Fourteen  limit  load  proof  test  conditions 
were  chosen  that  represented  selected  aircraft 
ground  handling,  landing,  braking,  flight  and 
weapon  launcher  conditions.  Some  test  load 
conditions  were  truncated  and  in  some  cases 
two  truncated  conditions  were  combined  and 
run  as  one  condition. 

A truncated  load  condition  is  one  that 
loads  only  the  vehicle  section  critical  for  a given 
condition.  Balance  loads  are  applied  elsewhere 
on  the  vehicle  for  reaction  purposes.  In  this  way 
the  total  truncated  condition  is  less  complex 
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than  if  a total  airplane  load  condition  were  ap- 
plied. Yet  assurance  of  structural  integrity  for 
the  critical  test  area  is  provided.  Two  truncated 
conditions  could  be  balanced  together  in  this 
same  manner. 

A large  number  of  strain  measuring  de- 
vices had  been  installed  during  manufacture  of 
the  aircraft.  Many  of  these  (called  “safety-of- 
flight”  gages)  were  mounted  on  structural  ele- 
ments whose  failure  could  result  in  serious 
damage  to  the  aircraft.  The  location  of  these  de- 
vices was  identical  to  that  of  gages  installed  on 
Aircraft  1 and  the  design  verification  test  com- 
ponents. Response  from  the  gages  during  the 
application  of  test  loads  provided  verification  of 
internal  structural  load  distributions.  Too,  a 
data  interface  between  the  design  verification 
test  and  Aircraft  1 and  Aircraft  2 flight  tests 
was  provided.  During  flight  testing,  the  stress 
readings  could  be  compared  with  established 
"never-exceed”  limits. 

As  a demonstration  of  structural  integrity 
at  100  percent  of  design  limit  load,  the  test 
reached  a successful  conclusion.  Primary  struc- 
tural failures  did  not  occur.  Typical  aircraft 
skin  panel  buckling  did  occur  in  a wide  range  of 
components,  but  after  unloading,  the  buckles 
disappeared  without  leaving  evidence  of  load 
induced  permanent  set.  Further,  the  strain  gage 
data  generally  followed  the  analytical  predic- 
tions within  reasonable  bounds.  Thus,  the  over- 
all soundness  of  the  structural  analysis  and  de- 
sign approach  were  successfully  demonstrated. 

Structural  Deflection  Tests 

The  B-1  development  program  originally 
included  the  requirement  for  structural  influ- 
ence coefficient  testing  on  the  complete  air- 
plane. The  purpose  of  structural  influence  coef- 
ficient testing  was  to  substantiate  the  stiffness 
characteristics  that  are  used  in  the  flutter  ana- 
lyses, the  design  of  flutter  models  and  the  exter- 
nal loads  analyses.  A structural  influence  coef- 
ficient describes  the  deflection  of  the  structure 
at  a point  on  the  airplane  caused  by  loading  ap- 
plied at  that  and  at  other  points.  In  a structural 
influence  coefficient  test  the  load  is  applied  con- 
secutively at  a number  of  control  points,  while 


simultaneously  measuring  the  resulting  deflec- 
tions at  all  those  points.  For  the  B-1  this  would 
have  resulted  in  45 1 separate  load  applications 
and  up  to  143  deflection,  twist  angle  or  slope 
measurements  at  each  load  application. 


Review  of  the  aircraft  industry’s  experi- 
ence in  performing  structural  deflection  coeffi- 
cient testing  revealed  that  such  testing  is  often 
performed  on  components  such  as  pylons,  actu- 
ators or  control  surfaces  where  detailed  know- 
ledge of  the  stiffness  characteristics  is  impor- 
tant. Structural  deflection  coefficient  testing 
over  a complete  aircraft  is  never  performed  and 
doubt  was  expressed  that  the  accuracy  of  the  re- 
sults would  be  better  than  results  obtainable  by 
simpler  means. 


As  a consequence,  the  requirement  for 
structural  influence  coefficient  tests  was  deleted 
and  replaced  with  structural  deflection  tests. 
Structural  deflections  were  measured  at  all  the 
structural  deflection  coefficient  control  points 
during  the  application  of  two-limit  load  proof 
test  conditions.  The  measured  deflections  could 
be  compared  with  predicted  deflections  to  ob- 
tain indirect  evaluation  of  the  influence  coeffi- 
cients. Results  from  ground  vibration  testing 
provided  additional  information  used  in  this 
evaluation. 


Limit  Load  Rigidity  Tests 

Limit  load  rigidity  tests  were  required  to 
determine  if  changes  in  stiffness  occur  from  the 
application  of  limit  load.  These  tests  consist  of 
deflection  measurements  similar  to  the  struc- 
tural deflection  test,  but  at  a considerably  re- 
duced number  of  control  points.  The  deflection 
measurements  help  to  determine  the  degree  and 
character  of  any  nonlinearity  the  structure 
might  exhibit  at  limit  load  magnitudes.  Deflec- 
tion measurements  for  the  limit  load  rigidity 
test  were  performed  in  conjunction  with  the 
limit  load  proof  tests  at  all  the  applied  test  con- 
ditions. The  results  showed  that  a reduction  in 
stiffness  does  not  occur  when  the  airframe  is 
subjected  to  limit  load. 
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Strain  Gage  Calibration  Tests 

The  primary  task  assigned  to  Aircraft  2 in- 
cluded the  structural  airloads/dynamic  re- 
sponse testing.  The  purpose  of  such  testing  is  to 
measure  in-flight  and  ground  operational  loads 
for  substantiation  of  the  structural  design  loads 
analyses.  Too,  structural  integrity  of  the  air- 
plane for  the  critical  loading  conditions  is  flight 
demonstrated. 

There  are  two  basic  methods  for  measur- 
ing structural  loads — the  strain  gage  method 
and  the  pressure  survey  method.  Both  methods 
provide  the  load  parameters  in  terms  of  bending 
moment,  shear  and  torsion  and  both  were  used 
on  the  B-1.  Figure  3 shows  the  distribution  of 
the  structural  airloads  instrumentation. 

The  use  of  strain  gages  for  measurement  of 
bending  moment,  shear  and  torsion  requires  a 
calibration.  This  calibration  is  to  provide  equa- 
tions in  terms  of  mathematical  combinations  of 
strain  gage  outputs  that  will  yield  values  of 
bending  moment,  shear  and  torsion.  The  most 
commonly  used  calibration  technique  requires 
the  application  of  incremental  loads  at  a num- 
ber of  individual  loading  points,  one  at  a time. 
This  point  load  method  of  calibration  is  not 
completely  satisfactory  since  it  calibrates  the 
system  for  only  a portion  of  the  loads  ultimately 
attained  in  actual  flight  test  operations.  A more 
ideal  way  of  performing  a calibration  is  through 
the  application  of  a series  of  distributed  loads 
representative  of  expected  loading  conditions. 
This  method  is  not  normally  used  because  it  re- 
quires a static  test  fixture  and  is  more  time  con- 
suming, complicated  and  costly. 

The  decision  to  submit  a B- 1 aircraft  to  a 
limit  load  proof  test  afforded  an  ideal  method  of 
calibration  using  distributed  loads  representa- 
tive not  only  of  actual  flight  conditions,  but  of 
predicted  critical  conditions  as  well.  Calibra- 
tion tests  were  run  in  conjunction  with  the  limit 
load  proof  tests  where  load  magnitudes  and 
center  of  pressure  locations  were  compatible. 
Additional  special  calibration  conditions,  not 
compatible  with  proof  test  conditions,  were  run 
separately  to  complete  the  calibration  test.  The 
loads  equations  derived  were  then  available  for 
use  in  the  flight  test  program  of  Aircraft  2. 


FLIGHT  TEST  PROGRAM 

Perhaps  the  most  visible  portion  of  aircraft 
development  is  the  flight  test  phase.  The  true 
test  of  an  aircraft  comes  when  it  is  called  upon 
to  fly  and  perform  as  designed.  From  a struc- 
tural integrity  standpoint,  design  performance 
has  less  visibility  during  flight  test.  The  struc- 
tural flight  test  program  seeks  to  measure  load 
distributions  and  approach,  but  not  reach,  aero- 
elastic  instability  points.  Success  is  measured  by 
a failure  to  find  higher  than  predicted  loadings, 
and  the  demonstration  of  adequate  structural 
damping  within  the  flight  envelope. 

Flight  Flutter  Testing 

Flight  flutter  tests  are  performed  to  sub- 
stantiate that  the  airplane  is  free  from  aeroelas- 
tic  instabilities  and  has  satisfactory  damping  up 
to  limit  speeds  of  the  flight  envelope.  This  is 
accomplished  through  the  definition  of  modal 
parameters,  principally  frequency  and  damp- 
ing, of  the  more  important  aeroelastic  vibration 
modes  of  the  aircraft  in  flight. 

To  accomplish  the  flight  flutter  tests,  flut- 
ter critical  substructures  on  the  aircraft  were 
forced  into  vibration  and  the  resulting  aeroelas- 
tic response  was  measured,  recorded  and  ana- 
lyzed. The  mechanism  for  forcing  the  vibration 
used  oscillating  inertia  shakers  that  were  in- 
stalled at  the  tips  of  the  wings  and  at  the  tips  of 
the  horizontal  and  vertical  stabilizers.  Excita- 
tion of  flight  vibration  modes  was  accomplished 
by  sweeping  the  frequency  range  from  1 to  60 
hertz  or  by  selection  of  a discrete  frequency. 
The  magnitude  of  the  shaker  inertia  forcing 
function  could  be  varied. 

The  Aircraft  1 flight  flutter  test  program 
covered  practicable  variations  in  parameters 
such  as  speed,  altitude  and  gross  weight.  Of 
particular  interest  was  the  effect  of  operation  of 
the  B-1  automatic  flight  control  (Stability  and 
Control  Augmentation)  and  terrain  following 
ride  quality  (Structural  Mode  Control)  systems. 
Flight  flutter  testing  for  flutter  clearance  of  the 
flight  envelope  is  performed  in  a “build-up” 
manner  and  must  precede  the  structural  flight 
loads  testing.  By  the  same  token.  Aircraft  1 is 
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limited  to  80  percent  of  design  limit  load  until 
completion  of  the  flight  loads  testing  to  1 00  per- 
cent of  design  limit  and  the  incorporation  of 
any  necessary  modifications  resulting  from  ei- 
ther the  flight  or  static  test  programs. 

Flight  flutter  data  were  compared  with 
theoretically  predicted  damping  trends  for  the 
more  important  structural  vibration  modes. 
When  practical  these  trends  were  used  to  dem- 
onstrate that,  even  with  an  increase  of  15  per- 
cent in  equivalent  airspeed  at  all  points  on  the 
limit  speed  envelope,  flutter  would  not  be  ex- 
pected. 

Flight  Loads  Testing 

Structural  flight  loads  tests  consist  of  flight 
and  ground  operation  load  surveys  and  dy- 
namic response  tests.  The  purpose  is: 

• to  substantiate  the  structural  loads 
analysis, 

• to  establish  the  variation  of  component 
loads  with  speed,  altitude  and  type  of 
maneuver  for  use  in  determining  crit- 
ical loading  conditions  and 

• to  demonstrate  aircraft  structural  in- 
tegrity for  these  critical  conditions. 

The  flight  and  ground  operations  load  sur- 
vey is  conducted  in  two  distinct  phases.  The  ini- 
tial phase  consists  of  a series  of  specified  load 
survey  maneuvers  to  80  percent  of  design  limit 
load.  The  test  conditions  cover  the  various  op- 
erating regimes  (gross  weight,  load  factor, 
speed,  altitude,  etc.)  so  that  the  measured  loads 
can  be  used  to  correlate  with  and  substantiate 
the  structural  loads  analysis.  Upon  completion 
of  the  initial  phase,  the  measured  loads  data  is 
extrapolated  to  100  percent  conditions  for  com- 
parison with  predicted  design  values  and  deter- 
mination of  critical  maneuver  conditions.  The 
final  phase  then  consists  of  build-up  maneuvers 
to  100  percent  of  design  limit  load  to  demon- 
strate the  conditions  which  were  indicated  as 
critical  by  analytical  calculations,  static  tests 
and  initial  flight  test  data. 


While  the  flight  and  ground  operations 
test  aircraft.  Aircraft  2,  proceeds  to  load  levels 
equal  to  design  limit,  other  test  aircraft  remain 
restricted  to  80  percent  load  levels.  The  release 
of  structural  operating  limitations  for  test  and 
service  inventory  aircraft  is  contingent  upon 
successful  completion  of  the  structural  certifi- 
cation program,  including  static  tests,  flight 
tests  and  structural  modifications  resulting 
therefrom. 


Dynamic  response  tests  are  performed 
concurrently  with  the  loads  survey  and  demon- 
stration tests.  These  tests  will  include  flight, 
landing  and  taxi  conditions  simulating  actual 
operations  to  verify  the  dynamic  loads  analysis 
and  response  characteristics.  Flights  will  be 
made  through  atmospheric  turbulence  for  the 
conditions  where  significant  dynamic  response 
is  anticipated.  Landing  tests  will  cover  a range 
of  sink  speeds,  and  taxi  tests  will  include  runs 
over  specific  runway  roughnesses  at  various 
speeds. 


The  load  measurements  obtained  from 
Aircraft  2 are  used  to  evaluate  the  adequacy  of 
the  loading  conditions  which  were  applied  in 
the  strength  tests  and,  if  necessary,  to  provide 
corrections  to  the  test  results  or  to  any  further 
ground  tests. 


In  addition  to  strength,  the  measured 
loads  data  is  particularly  relevant  to  the  estima- 
tion of  structural  fatigue  life.  Flight  measure- 
ments are  used  to  check  the  load  distributions 
and  the  frequency  of  occurrence  of  load  magni- 
tudes for  application  to  structural  fatigue  tests. 
Lessons  learned  during  flight  testing  must  be 
taken  into  account  in  the  fatigue  test  program  if 
structural  integrity  is  to  be  assured  throughout 
the  aircraft’s  service  life. 


At  this  writing,  the  initial  phase  of  the 
flight  load  survey  is  approximately  80  percent 
complete.  Results  of  the  testing  accomplished 
have  shown  the  measured  loads  to  be  equal  to 
or  less  than  predicted  for  the  primary  structural 
components. 
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FATIGUE  TEST  PROGRAM 

The  structural  life  discipline  has  made 
more  rapid  changes  in  recent  years  than  has  any 
area  of  structures  technology.  In  1970  when  the 
B-1  contract  was  awarded,  Air  Force  structural 
requirements  included  a “safe-life”  fatigue  de- 
sign using  a scatter  factor  of  four.'  This  factor 
was  assumed  to  account  for  the  effects  of  initial 
structural  quality,  environment  and  variation 
in  material  properties.  Fracture  mechanics  re- 
quirements had  not  been  defined  and  fatigue 
testing  could  be  done  on  a production  aircraft 
using  a “block”  loading  spectrum.  In  1974-75 
the  Air  Force  requirements  were  changed  to  the 
current  version.*  Fracture  mechanics  design 
and  testing  were  made  requirements  along  with 
fatigue  testing  prior  to  production  commitment 
using  a representative  flight-by-flight  loading 
spectrum. 

The  B-1  program  occurred  in  the  middle 
of  this  evolution  in  structural  philosophy  and 
helped  to  pave  the  way  for  the  present  day  re- 
quirements. The  B-1  was  the  first  Air  Force 
program  in  which  fracture  mechanics  was  insti- 
tuted as  a design  requirement  and  fracture  con- 
trol was  made  part  of  the  development  process.* 
Also,  the  B-1  program  was  the  first  program  to 
employ  major  fatigue  design  verification  test 
articles,  the  successful  testing  of  which  was  a 
prerequisite  to  DSARC  III. 

The  requirement  to  achieve  early  fatigue 
test  results  carried  with  it  some  consequent 
penalties.  The  B-1  DSARC  III  was  scheduled 
for  November  1976.  The  schedule  required  fa- 
tigue testing  to  commence  by  February  1976  on 
the  major  fatigue  design  verification  test  arti- 
cles, or  completion  of  manufacturing  by  the  end 
of  1975.  Design  release  of  hardware  drawings 
was  needed  approximately  2 years  ahead  of  this 
date  to  meet  material  procurement  and  manu- 
facturing schedules.  Thus,  the  design  of  the  fa- 
tigue design  verification  test  articles  had  to  be 
completed  in  the  1973  time  period,  before  any 
B-1  had  flown  and  when  there  were  still  many 
unknowns  about  the  fatigue  loading  spectrum. 

The  B-1  structural  design  loads  were  re- 
leased in  late  1972  and  early  1973.  They  were 


derived  for  a B-1  which  was  still  very  much  on 
paper.  Preliminary  weights  and  inertias  had 
been  iterated  to  an  extent,  structural  influence 
coefficients  were  based  on  general  internal  siz- 
ings and  some  wind  tunnel  data  was  available. 
However,  structural  dynamics  had  not  been  in- 
cluded and  only  a preliminary  description  of 
the  B- 1 stability  and  control  augmentation  sys- 
tem had  been  made. 

As  the  time  approached  to  begin  design 
verification  testing,  more  information  had  been 
obtained.  Revised  mass  distributions  and  iner- 
tias had  been  calculated,  along  with  updated  in- 
fluence coefficients.  Structural  dynamic  effects 
were  available  for  the  vertical  and  lateral  axes. 
Aerodynamic  definition  had  been  refined  such 
that  vertical  and  horizontal  tail  interference  ef- 
fects were  predicted.  As  a result  of  information 
gained  from  the  flight  control  simulator,  the 
importance  of  the  stability  and  control  augmen- 
tation system  to  the  B-1  fatigue  loading  spec- 
trum was  becoming  clear.  Finally,  weight 
growth  over  specification  goals  had  taken  place 
that  needed  to  be  offset  by  changes  in  opera- 
tional gross  weights. 

The  program  was  faced  with  the  choice  of 
updating  the  design  of  the  test  articles  to  reflect 
these  latest  inputs  or  testing  the  original  design 
to  higher  than  design  loads.  In  view  of  the 
disparities  known  to  exist  in  fatigue  analysis 
and  design  techniques,  the  decision  was  made 
to  proceed  with  testing  of  the  as-designed  arti- 
cles using  a fatigue  loading  spectrum  derived 
with  the  latest  loads  information.  In  this  way 
the  test  itself  would  provide  information  about 
areas  needing  redesign  rather  than  relying 
solely  on  analytical  predictions  for  final  struc- 
tural sizings. 

The  decision  to  update  the  structural  loads 
for  test  purposes  was  proved  sound  when  the  B- 
1 entered  flight  test.  Measured  load  levels  and 
frequency  of  occurrence  of  load  magnitudes 
showed  excellent  correlation  with  the  revised 
loading  used  during  design  verification  testing. 
That  revised  loading  spectrum  was  nevertheless 
more  severe  than  the  one  to  which  the  structure 
had  been  designed.  This  increased  the  risk  of 
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structural  cracking  and  failures  in  the  test  arti- 
cles, but  a fully  representative  test  spectrum 
was  deemed  important  enough  to  offset  risks  to 
the  test  schedule,  especially  in  view  of  the  crit- 
icality of  the  structure  being  tested. 

The  advantage  of  design  verification  test- 
ing is  that  large  scale  component  assemblies  can 
be  built  quicker  and  cheaper  than  can  an  entire 
airframe  thus  allowing  test  results  to  be  ob- 
tained earlier.  However,  some  rationale  must  be 
available  to  select  which  sections  of  the  aircraft 
will  be  tested  in  design  verification  test  and 
which  will  wait  for  a later  time.  In  the  case  of 
the  B-l  determination  of  critical  structure  for 
the  fatigue  design  verification  test  was  made  on 
the  basis  of  the  following: 

• Stress  levels  (life  assessment) 

• Flight  impact  if  failure  occurs  (loss  of 
aircraft,  aborted  mission,  complete 
mission) 

• Complexity  (load  paths,  advanced 
state  of  the  art) 

• Cost  impact  to  repair  (patch  vs  reskin) 

• Historical  problem  areas  (F-111,  C-5, 
etc.) 

The  design  verification  tests  were  scheduled  so 
results  obtained  could  be  incorporated  into  the 
production  design  and  avoid  expensive  struc- 
tural retrofit. 

In  view  of  the  above,  none  of  the  cracking 
that  finally  occurred  during  the  course  of  test- 
ing came  as  any  surprise;  that  was  the  reason 
for  the  tests.  In  every  case  where  cracking  oc- 
curred, design  improvements  were  incorpo- 
rated into  Aircraft  4,  the  test  article  was  re- 
paired and  testing  continued. 

Fatigue  testing  on  the  B-l  was  made  up  of 
two  phases — durability  testing  and  damage  tol- 
erance testing.  In  the  first  phase,  which  ran  for 
two  fatigue  lifetimes,  cracks  developed  natu- 
rally during  the  course  of  testing,  identifying 


structural  hot  spots.  These  cracks  were  for  the 
most  part  of  the  nuisance  variety  for  which  mi- 
nor detail  changes  were  generally  sufficient.  In 
the  damage  tolerance  phase  that  followed, 
small  artificial  flaws  were  installed  at  various 
locations  in  critical  primary  structural  areas. 
The  rate  of  growth  of  these  cracks  under  fatigue 
loading  could  then  be  used  to  assess  the  remain- 
ing structural  integrity  in  the  presence  of  unde- 
tected cracks  of  the  type  that  might  occur  in  ac- 
tual service.  This  remaining  structural  integrity 
is  called  damage  tolerance. 

The  most  ironic  observation  to  be  made 
about  the  results  of  the  fatigue  design  verifica- 
tion tests  deals  with  the  disparities  in  fatigue 
analysis  methods.  The  structural  areas  which 
were  of  most  concern  from  a fatigue  life  stand- 
point (e.g.,  wing  lower  surface,  wing  pivot  lugs 
and  fuselage  longeron  joints)  exhibited  almost 
no  cracking  problems.  The  areas  that  cracked 
were  largely  not  safety  critical  parts  and  in- 
cluded shear  webs,  fuselage  skins,  flanges  and 
fittings.  Little  attention  is  devoted  to  these 
types  of  parts  in  fatigue  literature  and  so  test- 
ing, rather  than  analysis,  must  be  used  to  iden- 
tify them  as  potential  problem  areas.  This  type 
of  cracking,  while  not  a safety  problem,  might 
have  posed  a maintenance  and  inspection  bur- 
den in  service  had  testing  not  identified  this 
cracking  for  redesign.  Therefore,  in  addition  to 
the  primary  purposes  of  eliminating  safety  of 
flight  problems  and  structural  recovery  pro- 
grams, the  fatigue  test  program  may  bring 
about  a reduction  in  cumulative  structural  in- 
spection and  repair  costs. 

CONCLUDING  REMARKS 

This  brief  discussion  of  the  B-l  full  scale 
structural  test  program  has  shown  how  the 
static,  fatigue  and  flight  test  elements  are  inter- 
related and  focus  on  a common  goal  of  assuring 
the  structural  integrity  throughout  the  useful 
life  of  the  aircraft. 

As  part  of  the  B-l  structural  development 
many  other  tests  were  performed  that  have  not 
been  discussed.  More  than  3000  material  allow- 
ables coupons,  1700  fracture  mechanics  allow- 
ables elements  and  approximately  700  design 


development  test  components  were  run  to  sup- 
port the  design  development  process.  More 
than  3000  hours  of  flutter  model  testing  to  sub- 
stantiate flutter  analyses  and  flutter  margins, 
and  approximately  5600  hours  of  wind  tunnel 
testing  to  define  aerodynamic  force  characteris- 
tics and  pressure  distributions  were  conducted. 
These  small  scale  development  tests  contrib- 
uted in  large  measure  to  the  eventual  successful 
progress  of  the  full-scale  structural  testing  pro- 
gram. 


The  primary  lesson  learned  during  the  B- 1 
structural  development  was  the  need  for  contin- 
ual interaction  between  the  various  engineering 
disciplines  involved.  Structural  integrity  is  the 
result  of  interdisciplinary  trade  off  and  opti- 
mization. The  structural  testing  elements  as 
performed  on  the  B-1  reflect  this  interrelation- 
ship and  thus  support  the  attainment  of  a qual- 
ity engineering  product  to  a stage  where  quality 
is  left  less  to  chance  and  more  to  the  integration 
of  disciplined  design  and  testing  practices. 


REFERENCES 


1.  Harold  M.  Wells,  and  Troy  T.  King,  “Air  Force  Aircraft  Structural  Integrity  Program:  Airplane  Requirements,” 

ASD-TR-66-57,  May  70. 

2.  Dept  of  Air  Force,  MIL-STD-1530A  (11),  “Aircraft  Structural  Integrity  Program,  Airplane  Requirements,”  II 

Dec  75. 

3.  Mark  A.  Owen,  “Evolution  of  Fracture  Control  in  the  B-1  Program,”  American  Society  for  Metals— American 

Society  of  Nondestructive  Testing  (ASM-ASNT)  Fifth  Annual  Forum:  Prevention  of  Failure  in  the  Transporta- 
tion Industry,  Tarpon  Springs,  FL,  Jun  1977. 


Mr.  John  W.  Rusten- 
burg  is  an  Aerospace  Engi- 
neer in  the  Systems  Project 
Office  of  the  Aeronautical 
Systems  Division,  Wright 
Patterson  Air  Force  Base, 

Ohio.  Mr,  Rustenburg  has 
been  with  the  B-1  Program 
throughout  its  development  phase  and  has  authored  a 
number  of  technical  reports  and  papers.  Prior  experience 
includes  the  XB-70  and  the  C-5A  Programs, 

Mr.  Rustenburg  received  a BS  (Aerospace  Engineer- 
ing) from  West  Virginia  University  in  1959. 


160 


Defense  Systems  Management  Review 


Mr.  Mark  A.  Owen  is 
an  Aerospace  Engineer  in 
the  Directorate  of  Flight 
Systems  Engineering,  Aero- 
nautical Systems  Division, 

Wright  Patterson  Air  Force 
Base,  Ohio.  Mr.  Owen  has 
been  with  the  B-1  Program 
throughout  its  development  phase  and  has  authored  a 
number  of  technical  reports  and  papers.  Prior  experience 
includes  the  XB-70  and  the  C-5A  Programs. 

Mr.  Owen  received  a BS  (Aerospace  Engineering) 
from  Purdue  in  1971.  He  received  his  MS  (Aerospace  En- 
gineering) from  the  Ohio  State  University  in  1977. 


Mr.  William  E.  Geese  is 
an  Aerospace  Engineer  in 
the  Strategic  Systems  Project 
Ofllce  of  the  Aeronautical 
Systems  Division,  Wright 
Patterson  Air  Force  Base, 

Ohio.  Mr.  Geese  has  been 
with  the  B-1  Program 
throughout  its  development  phase  and  has  authored  a 
number  of  technical  reports  and  papers.  Prior  experience 
includes  the  XB-70  and  C-5A  Programs. 

Mr.  Geese  received  a BS  (Aerospace  Engineering) 
from  the  University  of  Cincinnati  in  1969. 


Vol  I,  No.  5 


161 


PRODUCT  LIABILITY 


by 

Lt  Col  Robert  B.  Machen,  United  States  Army 

The  basis  for  this  thought  provoking  article  by  Lieutenant  Colonel  Robert  B.  Machen  was  a 
short  article  written  by  Mr.  Timothy  Kellerher.*  The  material  has  been  amplified  and  adapted 
by  Lieutenant  Colonel  Machen  to  stimulate  the  thoughts  of  /?eriVir  readers. 

Beyond  the  introduction  Lieutenant  Colonel  Machen  provides  a discussion  that  penetrates 
the  present  confusion  surrounding  reliability,  responsibility  and  redress — three  R’s  that  have  a 
direct  relation  to  the  Government’s  own  test  and  evaluation  of  military  equipment. 

Sharp  insights  are  shared  with  the  reader  who  will  gain  an  enhanced  awareness  of  the 
implication  of  products  defects.  The  policy  maker,  the  systems  acquisition  expert  and  the 
Program  Manager  alike  will  find  this  discussion  of  product  liability  informative,  stimulating 
and  most  importantly,  beneficial. 


“The  Court  of  the  Town  of  Williamsburg, 
County  of  York,  is  now  in  session.”  Let’s  imag- 
ine the  town  crier  of  1782  giving  notice  of  the 
case  of  Jonathan  Thatcher  vs  Thomas  Wyth, 
gunsmith.  Jonathan’s  militiaman  son,  Isaac, 
lost  a hand  in  an  accident  with  a musket  and 
Jonathan  is  suing  for  the  loss  of  the  son’s  hand, 
the  pain  and  suffering  endured,  disfigurement, 
and  loss  of  future  employment  in  the  militia. 
Thomas  admits  that  he  made  and  sold  the 
musket  in  1759  and  argues  that  the  gun  had 
changed  hands  many  times  before  it  was  ac- 
quired by  Isaac  as  an  issue  item  of  the  Com- 
monwealth. The  Commonwealth  admits  that 
the  stock  had  been  replaced  and  the  sights 
adjusted  several  times  by  unknown  users  while 
the  musket  was  out  of  the  control  of  the  gun- 
smith for  more  than  20  years  from  the  time  of 
manufacture.  Thomas  also  argued  that  he  had 
made  all  his  muskets  for  squirrel  hunting  and 
had  not  made  instruments  of  war,  but  the 
lawyer  for  Isaac  insisted  that  the  basic  design 
was  wrong  and  that  the  hammer  should  not 
have  been  sharp  on  one  side.  If  the  gunsmith 


•Timothy  Kellerher,  “Hear  Ye.  Hear  Ye,”  Alexandrian, 
Vol  3 (11):  23  (Nov  77). 


had  simply  rounded  both  sides  of  the  hammer, 
the  hammer  could  not  have  cut  the  thumb  of 
Jonathan’s  left-handed  son  and  caused  an  infec- 
tion, which  led  to  loss  of  the  hand.  The  lawyer 
was  very  persuasive  in  his  argument  that  the 
gunsmith  should  have  known  that  the  equip- 
ment would  be  used  in  an  environment  where 
infection  was  likely,  and  arguably,  following  a 
certain  tea  party  and  in  retrospect,  it  was  also 
foreseeable  that  the  colonists  could  rebel 
against  the  Crown  before  the  musket  would 
wear  out.  The  lawyer  showed  that  the  chain  of 
events  were  crystal  clear  to  those  who  would 
only  look. 

In  vain  did  Thomas  protest  that  Thatcher 
had  already  won  his  negligence  suit  against  Dr. 
O’Neal,  who  failed  to  save  the  hand.  The  results 
of  that  suit  had  driven  the  physician  to  bank- 
ruptcy. The  law  was  straightforward.  The  man- 
ufacturer was  responsible  for  his  product.  The 
passage  of  time  did  not  make  a difference.  The 
gunsmith  had  to  pay  for  his  poor  design. 

Consider  the  more  recent  history  of  pro- 
duct liability  in  the  United  States  starting  with 
the  landmark  decision  in  MacPherson  v.  Buick 
Motor  Company.  Justice  Cardoza’s  decision 
handed  down  in  New  York  in  1916  was  one  of 


162 


Defense  Systems  Minagement  Review 


the  first  major  steps  away  from  what  is  called 
privity  of  contract;  that  is,  the  governing  rela- 
tionship between  two  people  in  contracts  ex- 
cluding any  interest  of  a third  person  regardless 
of  impact.  Cardoza’s  turn  away  from  the  pro- 
tection afforded  by  privity  toward  strict  liabili- 
ty in  the  interest  of  third  parties  opened  an  en- 
tirely new  field  of  law  in  cases  against  manufac- 
turers. Simply  stated,  MacPherson  bought  a 
new  car  from  the  Buick  Mo  ' Company  who 
manufactured  automobiles.  The  car  was  sold  to 
a retailer  who  sold  it  to  MacPherson.  While 
MacPherson  was  driving  the  car,  a wheel  col- 
lapsed and  he  was  thrown  out  and  injured.  One 
of  the  spokes  was  made  of  defective  wood.  Al- 
though Buick  had  bought  the  wheel  from 
another  manufacturer,  it  was  shown  that  the 
defects  could  have  been  discovered  by  simple 
inspection.  The  holding  that  the  manufacturer 
was  responsible  for  the  finished  product,  even 
though  the  product  was  processed  through  a re- 
tailer to  a customer,  is  known  to  almost  every- 
one who  studies  law.  What  is  not  commonly 
known,  however,  is  that  there  has  been  a virtual 
explosion  of  court  decisions  during  the  seven- 
ties favoring  the  user.  It  stands  to  reason  that 
the  surface  has  just  been  scratched,  for  the  Na- 
tional Safety  Council  estimated  that  in  1975 
over  thirty  million  injuries  were  related  to  pro- 
ducts defects.  The  explosion  of  litigation  can,  in 
part,  be  attributed  to  the  nonapplication  of  old 
standby  defenses,  such  as  contributory  negli- 
gence and  assumption  of  risk.  Both,  along  with 
the  defense  of  privity,  have  been  cast  aside  by 
recent  court  decisions. 


The  absurdity  of  the  situation  can  be 
shown  by  further  paraphrasing  of  the  article 
referenced.  The  facts  reveal  the  following  in  a 
recent  suit  against  “Old  Saw  Manufacturing 
Co.”  In  1942,  some  35  years  ago,  the  company 
made  a bench  saw  with  the  customary  safety 
guards.  The  saw  was  delivered  under  wartime 
contract  and  met  all  of  the  specifications  ap- 
plied by  the  government  to  such  purchases. 
Sometime  after  World  War  II,  when  it  was 
decided  that  the  saw  was  worn  and  surplus,  it 
was  sold.  The  saw  then  changed  hands  a num- 
ber of  times.  It  was  worn  out,  rebuilt  and,  after 
more  than  30  years,  all  of  the  saw  that  was 
recognizable  was  the  original  nameplate  “Old 


Saw  Manufacturing  Company.”  In  its  last  rest- 
ing place,  the  saw  was  set  up  without  the  safety 
guards  of  a bench  saw  and  used  by  personnel 
who  apparently  knew  little  of  the  tradeoffs 
between  convenience  of  work  and  the  danger  of 
an  exposed  blade. 

A man  was  hurt  and  went  for  relief.  The 
attorney  knew  that  the  small  operation  which 
was  using  a 30  year-old  saw  had  no  money,  so 
he  looked  up  the  manufacturer.  That’s  what  a 
good  lawyer  is  paid  for  in  our  present  culture. 
Sure  enough  the  Dunn  & Bradstreet  rating  was 
good,  so  “Old  Saw  Manufacturing  Company” 
was  sued!  Any  lawyer  that  smart  had  to  be 
successful.  He  was  able  to  obtain  a jury  trial  and 
the  “Old  Saw  Manufacturing  Company”  paid 
through  the  nose  for  a verdict  on  the  design  of 
the  old  saw. 

The  liability  of  manufacturers  is  approach- 
ing that  of  the  physician  as  product  liability  in- 
surance costs  increase  drastically.  The  costs  of 
product  liability  claims  rose  more  than  seven 
times  in  the  last  5 years.  One  manufacturer  who 
lost  a suit  had  his  liability  insurance  premiums 
increase  from  below  $5,000  in  1971  to  almost 
$500,000  in  1977. 

The  full  impact  of  the  drastic  changes  of 
the  past  few  years  has  not  become  apparent  to 
the  manufacturers  of  defense  products.  Only 
time  will  tell  whether  or  not  the  first  hypotheti- 
cal story  becomes  a reality.  One  only  has  to 
look  at  a weekly  summary  of  military  aircraft 
accident  reports  or  other  similar  summaries  to 
realize  that  products  fail  daily.  The  next  ques- 
tion to  be  asked  is.  When  will  the  parents  of  the 
young  soldiers,  sailors,  and  airmen  realize  the 
significance  of  product  liability?  Will  they  soon 
discover  that  b^ause  it’s  painted  Army  Green, 
Battleship  Gray,  or  Air  Force  Blue,  it  makes  no 
difference? 

How  does  the  foregoing  affect  the  policy 
makers  of  the  Armed  Forces,  the  Systems  Ac- 
quisition Managers  and  the  Program  Managers 
of  defense  systems?  Perhaps  the  most  telling 
impact  is  recognition  of  the  significance  of  man- 
ufacturer responsibility  and  how  legal  determi- 
nation of  product  liability  is  established. 
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The  program  Manager  who  nurtures  a 
major  program  from  inception  to  completion  is 
indeed  a fortunate  individual  if  the  task  is 
accomplished  without  the  disruptive  influences 
of  at  least  one  near  (or  actual)  catastrophic 
occurrence — an  occurrence  that  is  attributable 
to  defective  equipment  such  as  a component  or 
part. 

When  a product  malfunction  leads  to  ad- 
verse results  (injury  or  death  to  product  users) 
and  a program  goes  awry,  think  on  these  things 
(all  elements  of  a cause  of  action): 

• Was  there  an  equipment  (product)  de- 
fect? 

• Was  the  defect  present  when  the  pro- 
duct first  was  delivered? 


Again  if  either  one  or  a combination  of 
these  questions  are  affirmative  you  can  rest 
assured  that  you  are  in  for  a real  education — 
the  same  type  of  education  that  has  haunted  the 
medical  profession  for  the  past  few  years.  As 
you  work  toward  satisfying  test  and  evaluation 
requirements  consider  the  emphasis  you  have 
(or  have  not)  placed  on  the  quality  of  the 
products  that  comprise  the  system  of  your 
program. 

A reasonable  conclusion  is  that  manufac- 
turers will  pay  for  injuries  caused  by  defects  in 
their  products.  The  day  has  arrived  when  one 
can  safely  say  “caveat  vendor”  instead  of  “ca- 
veat emptor.” 


• Was  the  defect  the  proximate  cause  of 
the  injury? 

• Was  the  plaintiff  someone  that  the 
manufacturer  could  reasonably  foresee 
using  the  product  for  which  it  was 
manufactured  and  sold? 

In  the  event  that  either  or  all  of  the  above 
questions  are  positive,  the  next  step  will  be  to 
determine: 

1.  Is  there  a cause  of  action  based  on 
contract?  Such  a cause  of  action  re- 
quires that  there  be  a contractual  rela- 
tionship between  the  plaintiff  and 
defendimt. 

2.  Is  there  an  action  in  tort  which  is 
grounded  upon  negligence?  Such  a 
cause  of  action  does  not  require  the 
allegation  of  a contractual  relationship 
between  the  plaintiff  and  defendant. 

3.  Is  there  an  action  in  tort  which  is  based 
on  a breach  of  duty  assumed  by  the 
manufacturer  or  seller  of  a product 
when  the  manufacturer  expressed  or 
implied  a fitness  for  use  when  the 
pr^uct  was  delivered? 
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^ rk\J  I WITH  I ^ / / 


Page  vi,  line  5.  Reads:  Dr.  Albert  J.  Kelley,  President,  Author  D. 

Little  Program  Systems  Management  Company 
Should  read:  Dr.  Albert  J.  Kelley,  President,  Arthur 
D.  Little  Program  Systems  Management  Company 

J 

} 

Page  51,  column  1,  line  6.  Reads:  ...from  the  OMB-Base  Operations  budget;  \ 

Should  read:  ...from  the  OMA-Base  Operations  j 

budget;  j 

i 
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